[Scons-users] Recursive directories, and the ignoring of filenames.

[Marc Branchaud]

On 2018-08-02 07:45 PM, Alistair Buxton wrote:
> I want to build an openwrt image using their image builder. It is
> invoked like this:
> 
> make -C imagebuilder image FILES=/some/path
> 
> All files under /some/path will be inserted into the image. Therefore,
> if any file or subdirectory is renamed, the image needs to be rebuilt,
> ie the above command needs to be run again, even if the contents of
> the renamed file is the same as some previous iteration. It seems
> extraordinarily difficult to make scons do this, given how simple the
> concept is to explain.

I use os.walk() to make a big list of all the files.  The following 
assumes that the directory you want to walk is next to your SConscript.

	sources = []
	sources_dir = 'path/to/directory'
	for path, dirs, files in os.walk(sources_dir):
	    for f in files:
	        rel_path = os.path.join(path, f)[len(sources_dir)+1:]
	        sources.append(rel_path)

	env.Command(source = sources, .....)

(If you use variant directories, I find it simplifies things to make 
sources_dir an absolute path to the non-variant version of the directory.)

One nice thing about this approach is that I can selectively ignore 
paths I don't care about (e.g. ones that start with ".git/").

		M.


> Here is a simplified breakdown of the problem, using "ls" as a
> surrogate for the image builder, as it is simpler and everyone should
> be familiar with how it works:
> 
> I have a directory called 'files'. I want to produce a list of all the
> subdirectories and files in 'files' using scons and ls. A naive
> attempt would look like this:
> 
> env.Command('listing.txt', 'files', 'ls -lR ${SOURCE} > ${TARGET}')
> 
> The above will work exactly once, and as long as 'listing.txt' is
> never deleted, never again.
> 
> After some research you might come up with this:
> 
> env.Command('listing.txt', ['files/', Glob('files/*')], 'ls -lR
> ${SOURCES[0]} > ${TARGET}')
> 
> This will fail if 'files' has subdirectories, because Glob is not recursive.
> 
> After googling for "scons recursive glob" you might come up with
> something like this:
> 
> env.Command('listing.txt', ['files/', Glob('files/*'),
> Glob('files/*/*'), Glob('files/*/*/*'), Glob('files/*/*/*/*')], 'ls
> -lR ${SOURCES[0]} > ${TARGET}')
> 
> And you might even think it works... but it doesn't. Besides the
> problem that it will miss changes deeper than the number of globs you
> specify, it has a more subtle problem. If a file or directory is
> renamed, scons may or may not ignore it. You can demonstrate this by
> taking the above code, and putting it in a directory with
> 'files/a.txt' and 'files/b.txt'. Run scons, then rename b.txt to c.txt
> and re-run scons. The listing will not be updated, which is incorrect
> behaviour by any reasonable definition.
> 
> This happens because scons does not look at the filenames of
> dependencies. It assumes that they will be delivered in the same order
> each time, and only rebuilds if the number of items changed, or if the
> nth checksum does not match the previous nth checksum regardless of
> the filenames involved. As a result it can miss file and directory
> renames. It can also not if the renaming causes dependencies to be
> listed in a different order (ie if you renamed a.txt instead of b.txt,
> and the files had different contents). This is rather illogical as it
> means the problem won't happen every time you rename a file. Even
> knowing why it happens requires deep knowledge of scons implementation
> details that should be irrelevant to the user.
> 
> One possible workaround is to do this:
> 
> env.Command('listing.txt', ['files',
> Value(subprocess.check_output(['sh', '-c', 'find files/ -type f -exec
> sha256sum {} \; | sort | sha256sum']))], 'ls -lR ${SOURCES[0]} >
> ${TARGET}')
> 
> This handles any level of recursion in the directory and also won't
> miss directory renames (as long as the directory is not empty). It
> sure is ugly to look at though.
> 
> Apparently there is a way to do this with a custom decider, but I
> cannot make it work. Deciders receive two Nodes and a FileNodeInfo,
> none of which contains any information about the previous filename of
> a dependency, or the other dependencies of a target and what their
> previous filenames may have been:
> 
> env = Environment(tools=[])
> 
> def my_decider(dependency, target, info):
>      print(target.get_binfo().bdepends)
>      return True
> 
> env.Decider(my_decider)
> 
> env.Command('out.txt', Glob('files/*'), 'ls -lR files/ > ${TARGET}')
> 
> Outputs:
> 
> scons: Reading SConscript files ...
> scons: done reading SConscript files.
> scons: Building targets ...
> []
> []
> []
> ls -lR files/ > out.txt
> []
> []
> []
> scons: done building targets.
> 
> as such, a decider can't possibly know if the list of dependencies has changed.
> 
> So in summary, implementing recursive glob alone is not enough. Scons
> also needs to take care of filename changes in order for recursive
> glob to be useful.
>