[Scons-users] Recursive directories, and the ignoring of filenames.

[Alistair Buxton]

PS note that the directory isn't even needed to reproduce the renaming
problem. The following will have the same behaviour if the a.txt and
b.txt are in the same directory as SConstruct:

env.Command('listing.txt', Glob('*.txt'), 'ls *.txt > ${TARGET}')

In this case you would instead say:

env.Command('listing.txt', Glob('*.txt'), 'ls ${SOURCES} > ${TARGET}')

and scons would identify that the build needs to be re-done because
the action string changed.

However, this is not so easy with the image builder case as it takes a
directory path as the only argument.

On 3 August 2018 at 00:45, Alistair Buxton <a.j.buxton at gmail.com> wrote:
> I want to build an openwrt image using their image builder. It is
> invoked like this:
>
> make -C imagebuilder image FILES=/some/path
>
> All files under /some/path will be inserted into the image. Therefore,
> if any file or subdirectory is renamed, the image needs to be rebuilt,
> ie the above command needs to be run again, even if the contents of
> the renamed file is the same as some previous iteration. It seems
> extraordinarily difficult to make scons do this, given how simple the
> concept is to explain.
>
> Here is a simplified breakdown of the problem, using "ls" as a
> surrogate for the image builder, as it is simpler and everyone should
> be familiar with how it works:
>
> I have a directory called 'files'. I want to produce a list of all the
> subdirectories and files in 'files' using scons and ls. A naive
> attempt would look like this:
>
> env.Command('listing.txt', 'files', 'ls -lR ${SOURCE} > ${TARGET}')
>
> The above will work exactly once, and as long as 'listing.txt' is
> never deleted, never again.
>
> After some research you might come up with this:
>
> env.Command('listing.txt', ['files/', Glob('files/*')], 'ls -lR
> ${SOURCES[0]} > ${TARGET}')
>
> This will fail if 'files' has subdirectories, because Glob is not recursive.
>
> After googling for "scons recursive glob" you might come up with
> something like this:
>
> env.Command('listing.txt', ['files/', Glob('files/*'),
> Glob('files/*/*'), Glob('files/*/*/*'), Glob('files/*/*/*/*')], 'ls
> -lR ${SOURCES[0]} > ${TARGET}')
>
> And you might even think it works... but it doesn't. Besides the
> problem that it will miss changes deeper than the number of globs you
> specify, it has a more subtle problem. If a file or directory is
> renamed, scons may or may not ignore it. You can demonstrate this by
> taking the above code, and putting it in a directory with
> 'files/a.txt' and 'files/b.txt'. Run scons, then rename b.txt to c.txt
> and re-run scons. The listing will not be updated, which is incorrect
> behaviour by any reasonable definition.
>
> This happens because scons does not look at the filenames of
> dependencies. It assumes that they will be delivered in the same order
> each time, and only rebuilds if the number of items changed, or if the
> nth checksum does not match the previous nth checksum regardless of
> the filenames involved. As a result it can miss file and directory
> renames. It can also not if the renaming causes dependencies to be
> listed in a different order (ie if you renamed a.txt instead of b.txt,
> and the files had different contents). This is rather illogical as it
> means the problem won't happen every time you rename a file. Even
> knowing why it happens requires deep knowledge of scons implementation
> details that should be irrelevant to the user.
>
> One possible workaround is to do this:
>
> env.Command('listing.txt', ['files',
> Value(subprocess.check_output(['sh', '-c', 'find files/ -type f -exec
> sha256sum {} \; | sort | sha256sum']))], 'ls -lR ${SOURCES[0]} >
> ${TARGET}')
>
> This handles any level of recursion in the directory and also won't
> miss directory renames (as long as the directory is not empty). It
> sure is ugly to look at though.
>
> Apparently there is a way to do this with a custom decider, but I
> cannot make it work. Deciders receive two Nodes and a FileNodeInfo,
> none of which contains any information about the previous filename of
> a dependency, or the other dependencies of a target and what their
> previous filenames may have been:
>
> env = Environment(tools=[])
>
> def my_decider(dependency, target, info):
>     print(target.get_binfo().bdepends)
>     return True
>
> env.Decider(my_decider)
>
> env.Command('out.txt', Glob('files/*'), 'ls -lR files/ > ${TARGET}')
>
> Outputs:
>
> scons: Reading SConscript files ...
> scons: done reading SConscript files.
> scons: Building targets ...
> []
> []
> []
> ls -lR files/ > out.txt
> []
> []
> []
> scons: done building targets.
>
> as such, a decider can't possibly know if the list of dependencies has changed.
>
> So in summary, implementing recursive glob alone is not enough. Scons
> also needs to take care of filename changes in order for recursive
> glob to be useful.
>
> --
> Alistair Buxton
> a.j.buxton at gmail.com



-- 
Alistair Buxton
a.j.buxton at gmail.com