[Scons-users] Recursive directories, and the ignoring of filenames.

[Alistair Buxton]

I want to build an openwrt image using their image builder. It is
invoked like this:

make -C imagebuilder image FILES=/some/path

All files under /some/path will be inserted into the image. Therefore,
if any file or subdirectory is renamed, the image needs to be rebuilt,
ie the above command needs to be run again, even if the contents of
the renamed file is the same as some previous iteration. It seems
extraordinarily difficult to make scons do this, given how simple the
concept is to explain.

Here is a simplified breakdown of the problem, using "ls" as a
surrogate for the image builder, as it is simpler and everyone should
be familiar with how it works:

I have a directory called 'files'. I want to produce a list of all the
subdirectories and files in 'files' using scons and ls. A naive
attempt would look like this:

env.Command('listing.txt', 'files', 'ls -lR ${SOURCE} > ${TARGET}')

The above will work exactly once, and as long as 'listing.txt' is
never deleted, never again.

After some research you might come up with this:

env.Command('listing.txt', ['files/', Glob('files/*')], 'ls -lR
${SOURCES[0]} > ${TARGET}')

This will fail if 'files' has subdirectories, because Glob is not recursive.

After googling for "scons recursive glob" you might come up with
something like this:

env.Command('listing.txt', ['files/', Glob('files/*'),
Glob('files/*/*'), Glob('files/*/*/*'), Glob('files/*/*/*/*')], 'ls
-lR ${SOURCES[0]} > ${TARGET}')

And you might even think it works... but it doesn't. Besides the
problem that it will miss changes deeper than the number of globs you
specify, it has a more subtle problem. If a file or directory is
renamed, scons may or may not ignore it. You can demonstrate this by
taking the above code, and putting it in a directory with
'files/a.txt' and 'files/b.txt'. Run scons, then rename b.txt to c.txt
and re-run scons. The listing will not be updated, which is incorrect
behaviour by any reasonable definition.

This happens because scons does not look at the filenames of
dependencies. It assumes that they will be delivered in the same order
each time, and only rebuilds if the number of items changed, or if the
nth checksum does not match the previous nth checksum regardless of
the filenames involved. As a result it can miss file and directory
renames. It can also not if the renaming causes dependencies to be
listed in a different order (ie if you renamed a.txt instead of b.txt,
and the files had different contents). This is rather illogical as it
means the problem won't happen every time you rename a file. Even
knowing why it happens requires deep knowledge of scons implementation
details that should be irrelevant to the user.

One possible workaround is to do this:

env.Command('listing.txt', ['files',
Value(subprocess.check_output(['sh', '-c', 'find files/ -type f -exec
sha256sum {} \; | sort | sha256sum']))], 'ls -lR ${SOURCES[0]} >
${TARGET}')

This handles any level of recursion in the directory and also won't
miss directory renames (as long as the directory is not empty). It
sure is ugly to look at though.

Apparently there is a way to do this with a custom decider, but I
cannot make it work. Deciders receive two Nodes and a FileNodeInfo,
none of which contains any information about the previous filename of
a dependency, or the other dependencies of a target and what their
previous filenames may have been:

env = Environment(tools=[])

def my_decider(dependency, target, info):
    print(target.get_binfo().bdepends)
    return True

env.Decider(my_decider)

env.Command('out.txt', Glob('files/*'), 'ls -lR files/ > ${TARGET}')

Outputs:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
[]
[]
[]
ls -lR files/ > out.txt
[]
[]
[]
scons: done building targets.

as such, a decider can't possibly know if the list of dependencies has changed.

So in summary, implementing recursive glob alone is not enough. Scons
also needs to take care of filename changes in order for recursive
glob to be useful.

-- 
Alistair Buxton
a.j.buxton at gmail.com