[Scons-users] collecting groups of files as sources

[Gabe Black]

Hi, I'm a core developer on large project which uses SCons as our build
system (gem5.org), and I'm in the process of refactoring our build so that
it has less custom code and defers more work to SCons itself.

The biggest sticking points so far have generally been around gathering up
collections of source files to be built into an executable. This problem
crops up in at least two different flavors.


** Lists of sources **

The first, simplest version is that our project is quite large and has
many, many source files. It would be very impractical to try to put them
all in a huge list to supply to a Program builder, and this would also
architect out modularity of our build system, where source files would no
longer be handled locally and would instead have to be handled all in one
central clearing house. This would also break a mechanism we have which
lets users add new collections of source to the project to add their own
components. An approximate analogy would be kernel modules. It would not be
possible to add these extra directories without modifying the main projects
source if you had to, by definition, modify the huge list of source files
to extend the code base.

To solve this problem, we currently declare python objects which represent
the source files, but don't actually mean anything to SCons itself. These
objects also carry "tags" so that they can say they should only be included
if the project includes its built in python interpreter, or for certain
unit tests, or if it's being built as a complete executable instead of as a
library.

Then later, after all the SConscript files in subdirectories have be
processed, we have a mechanism to generate lists of source files with
and/or without various tags, and then we feed those into the actual
Program, SharedLibrary, or StaticLibrary builders.

I would really rather have something more like this:

Program('foo.bin', '${SOURCES.with_any_tags("main", "lib", "python")')

and have that construction variable be expanded *after* all the sources
have been collected. It's relatively easy to centralize the declaration of
our central binary or libraries, but it's not really possible for things
like unit tests which are scattered throughout the code base (near what
they test) and can even come in through user additions.


** Dependencies based on build products **

Another more complicated problem is that our project is a simulator, and
the objects in the simulator are described using python. Our build actually
imports these modules, and then checks to see which different classes of
simulation objects have been set up. Then each of these are used to
generate additional c++ files which act as the glue between those python
classes and the c++ classes that underlie them.

The problem here is similar, in that we need to collect all the python
modules, import them and generate a list of simulation objects, and then
based on that generate a collection of .cc files which will be built into
the simulator. We need to collect the .py files which is like above, but
then beyond that we need to run a build step, and then based on what
happens there add some number of .cc files to the build.

What we're doing now is that we just have a step in the SConscript which
does all that in line, and then we set up additional source file
representing objects like I described above. This adds a decent amount of
complex custom code to our build scripts, and another sequential element to
the build process.

Is there a way I can run the build step of importing all these .py, and
then add the extra .cc files to the build? I wasn't able to think of any
way to do that given how scons works, and this Stack Overflow post was the
closest I've found from anybody else, although it looks like it's abusing
internal interfaces and I'm pretty reluctant to do anything like that in
our production code:

https://stackoverflow.com/questions/24671859/scons-how-to-generate-dependencies-after-some-targets-have-been-built

Are scanners the right way to do this somehow? Does SCons have any sort of
mechanism where it can re-scan nodes that have changed since the build
started?


** Multiple invocations? **

Another big hammer approach I'm considering is to write some sort of
wrapper script which will just invoke SCons multiple times, once for each
layer of dependencies in the build, and have it record the intermediate
results someplace for it rediscover between runs.

This feels pretty clunky to me, and like something SCons should be handling
for me. Is there a better way?

Gabe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210724/01f0f831/attachment.htm>