[Scons-users] Recursive directories, and the ignoring of filenames.

[Jack Brennen]

For your first example, which you say will work exactly once, and never 
again...

Can you just make that node dependent on a Value node with the tuple of 
the current time and PID as the Value?

import os, time
env.Depends('listing.txt', env.Value((time.time(),os.getpid())))


By making it dependent on the tuple of time.time() with os.getpid(), it 
should effectively rerun the "ls" command every time you execute SCons.  
The chance of getting the same time and the same PID in different SCons 
executions should be vanishingly small.

- Jack

On 8/2/2018 4:45 PM, Alistair Buxton wrote:
> I want to build an openwrt image using their image builder. It is
> invoked like this:
>
> make -C imagebuilder image FILES=/some/path
>
> All files under /some/path will be inserted into the image. Therefore,
> if any file or subdirectory is renamed, the image needs to be rebuilt,
> ie the above command needs to be run again, even if the contents of
> the renamed file is the same as some previous iteration. It seems
> extraordinarily difficult to make scons do this, given how simple the
> concept is to explain.
>
> Here is a simplified breakdown of the problem, using "ls" as a
> surrogate for the image builder, as it is simpler and everyone should
> be familiar with how it works:
>
> I have a directory called 'files'. I want to produce a list of all the
> subdirectories and files in 'files' using scons and ls. A naive
> attempt would look like this:
>
> env.Command('listing.txt', 'files', 'ls -lR ${SOURCE} > ${TARGET}')
>
> The above will work exactly once, and as long as 'listing.txt' is
> never deleted, never again.
>
> After some research you might come up with this:
>
> env.Command('listing.txt', ['files/', Glob('files/*')], 'ls -lR
> ${SOURCES[0]} > ${TARGET}')
>
> This will fail if 'files' has subdirectories, because Glob is not recursive.
>
> After googling for "scons recursive glob" you might come up with
> something like this:
>
> env.Command('listing.txt', ['files/', Glob('files/*'),
> Glob('files/*/*'), Glob('files/*/*/*'), Glob('files/*/*/*/*')], 'ls
> -lR ${SOURCES[0]} > ${TARGET}')
>
> And you might even think it works... but it doesn't. Besides the
> problem that it will miss changes deeper than the number of globs you
> specify, it has a more subtle problem. If a file or directory is
> renamed, scons may or may not ignore it. You can demonstrate this by
> taking the above code, and putting it in a directory with
> 'files/a.txt' and 'files/b.txt'. Run scons, then rename b.txt to c.txt
> and re-run scons. The listing will not be updated, which is incorrect
> behaviour by any reasonable definition.
>
> This happens because scons does not look at the filenames of
> dependencies. It assumes that they will be delivered in the same order
> each time, and only rebuilds if the number of items changed, or if the
> nth checksum does not match the previous nth checksum regardless of
> the filenames involved. As a result it can miss file and directory
> renames. It can also not if the renaming causes dependencies to be
> listed in a different order (ie if you renamed a.txt instead of b.txt,
> and the files had different contents). This is rather illogical as it
> means the problem won't happen every time you rename a file. Even
> knowing why it happens requires deep knowledge of scons implementation
> details that should be irrelevant to the user.
>
> One possible workaround is to do this:
>
> env.Command('listing.txt', ['files',
> Value(subprocess.check_output(['sh', '-c', 'find files/ -type f -exec
> sha256sum {} \; | sort | sha256sum']))], 'ls -lR ${SOURCES[0]} >
> ${TARGET}')
>
> This handles any level of recursion in the directory and also won't
> miss directory renames (as long as the directory is not empty). It
> sure is ugly to look at though.
>
> Apparently there is a way to do this with a custom decider, but I
> cannot make it work. Deciders receive two Nodes and a FileNodeInfo,
> none of which contains any information about the previous filename of
> a dependency, or the other dependencies of a target and what their
> previous filenames may have been:
>
> env = Environment(tools=[])
>
> def my_decider(dependency, target, info):
>      print(target.get_binfo().bdepends)
>      return True
>
> env.Decider(my_decider)
>
> env.Command('out.txt', Glob('files/*'), 'ls -lR files/ > ${TARGET}')
>
> Outputs:
>
> scons: Reading SConscript files ...
> scons: done reading SConscript files.
> scons: Building targets ...
> []
> []
> []
> ls -lR files/ > out.txt
> []
> []
> []
> scons: done building targets.
>
> as such, a decider can't possibly know if the list of dependencies has changed.
>
> So in summary, implementing recursive glob alone is not enough. Scons
> also needs to take care of filename changes in order for recursive
> glob to be useful.
>