[Scons-users] collecting groups of files as sources

[Bill Deegan]

Gabe,
Welcome!

Gem5 is a long time SCons user.
I've helped a few of your users over the past few years as they ran into
various issue.
Any chance we can get you to join our discord server?
https://discord.gg/pejaFYrD9n

I've just cloned your repo to take a look.
I'm on vacation for a few days, but if you give me a bit of time, I'll be
able to make better suggestions.

On possibility comes to mind, any chance that you could segregate the
various targets into separate directories on a per program basis?
That would simplify to Glob('#/build/program1/objects/*.obj') for example.
(I'm guessing it wouldn't be this simple, but worth asking)

-Bill
SCons Project Co-Manager

On Sat, Jul 24, 2021 at 3:02 AM Gabe Black <gabe.black at gmail.com> wrote:

> Hi, I'm a core developer on large project which uses SCons as our build
> system (gem5.org), and I'm in the process of refactoring our build so
> that it has less custom code and defers more work to SCons itself.
>
> The biggest sticking points so far have generally been around gathering up
> collections of source files to be built into an executable. This problem
> crops up in at least two different flavors.
>
>
> ** Lists of sources **
>
> The first, simplest version is that our project is quite large and has
> many, many source files. It would be very impractical to try to put them
> all in a huge list to supply to a Program builder, and this would also
> architect out modularity of our build system, where source files would no
> longer be handled locally and would instead have to be handled all in one
> central clearing house. This would also break a mechanism we have which
> lets users add new collections of source to the project to add their own
> components. An approximate analogy would be kernel modules. It would not be
> possible to add these extra directories without modifying the main projects
> source if you had to, by definition, modify the huge list of source files
> to extend the code base.
>
> To solve this problem, we currently declare python objects which represent
> the source files, but don't actually mean anything to SCons itself. These
> objects also carry "tags" so that they can say they should only be included
> if the project includes its built in python interpreter, or for certain
> unit tests, or if it's being built as a complete executable instead of as a
> library.
>
> Then later, after all the SConscript files in subdirectories have be
> processed, we have a mechanism to generate lists of source files with
> and/or without various tags, and then we feed those into the actual
> Program, SharedLibrary, or StaticLibrary builders.
>
> I would really rather have something more like this:
>
> Program('foo.bin', '${SOURCES.with_any_tags("main", "lib", "python")')
>
> and have that construction variable be expanded *after* all the sources
> have been collected. It's relatively easy to centralize the declaration of
> our central binary or libraries, but it's not really possible for things
> like unit tests which are scattered throughout the code base (near what
> they test) and can even come in through user additions.
>
>
> ** Dependencies based on build products **
>
> Another more complicated problem is that our project is a simulator, and
> the objects in the simulator are described using python. Our build actually
> imports these modules, and then checks to see which different classes of
> simulation objects have been set up. Then each of these are used to
> generate additional c++ files which act as the glue between those python
> classes and the c++ classes that underlie them.
>
> The problem here is similar, in that we need to collect all the python
> modules, import them and generate a list of simulation objects, and then
> based on that generate a collection of .cc files which will be built into
> the simulator. We need to collect the .py files which is like above, but
> then beyond that we need to run a build step, and then based on what
> happens there add some number of .cc files to the build.
>
> What we're doing now is that we just have a step in the SConscript which
> does all that in line, and then we set up additional source file
> representing objects like I described above. This adds a decent amount of
> complex custom code to our build scripts, and another sequential element to
> the build process.
>
> Is there a way I can run the build step of importing all these .py, and
> then add the extra .cc files to the build? I wasn't able to think of any
> way to do that given how scons works, and this Stack Overflow post was the
> closest I've found from anybody else, although it looks like it's abusing
> internal interfaces and I'm pretty reluctant to do anything like that in
> our production code:
>
>
> https://stackoverflow.com/questions/24671859/scons-how-to-generate-dependencies-after-some-targets-have-been-built
>
> Are scanners the right way to do this somehow? Does SCons have any sort of
> mechanism where it can re-scan nodes that have changed since the build
> started?
>
>
> ** Multiple invocations? **
>
> Another big hammer approach I'm considering is to write some sort of
> wrapper script which will just invoke SCons multiple times, once for each
> layer of dependencies in the build, and have it record the intermediate
> results someplace for it rediscover between runs.
>
> This feels pretty clunky to me, and like something SCons should be
> handling for me. Is there a better way?
>
> Gabe
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210725/5beff192/attachment-0001.htm>