[Scons-users] collecting groups of files as sources

[Gabe Black]

Thanks! Sure, I'll join your discord server! Please, take your time, your
suggestions are of course welcome. I'm in the midst of a deep
clean/refactoring of our build process, and you can find my pending changes
in this series in our code review system:

https://gem5-review.googlesource.com/c/public/gem5/+/48381

My current goal is to reduce the amount of custom code and extra machinery
we have, and push the work into SCons itself. In addition to making things
simpler in general, it will hopefully also make it easier to work on since
people will just need to know how SCons works, not how SCons works and how
our extra layers work on top of it.

Gabe

On Sun, Jul 25, 2021 at 8:35 PM Bill Deegan <bill at baddogconsulting.com>
wrote:

> Gabe,
> Welcome!
>
> Gem5 is a long time SCons user.
> I've helped a few of your users over the past few years as they ran into
> various issue.
> Any chance we can get you to join our discord server?
> https://discord.gg/pejaFYrD9n
>
> I've just cloned your repo to take a look.
> I'm on vacation for a few days, but if you give me a bit of time, I'll be
> able to make better suggestions.
>
> On possibility comes to mind, any chance that you could segregate the
> various targets into separate directories on a per program basis?
> That would simplify to Glob('#/build/program1/objects/*.obj') for example.
> (I'm guessing it wouldn't be this simple, but worth asking)
>
> -Bill
> SCons Project Co-Manager
>
> On Sat, Jul 24, 2021 at 3:02 AM Gabe Black <gabe.black at gmail.com> wrote:
>
>> Hi, I'm a core developer on large project which uses SCons as our build
>> system (gem5.org), and I'm in the process of refactoring our build so
>> that it has less custom code and defers more work to SCons itself.
>>
>> The biggest sticking points so far have generally been around gathering
>> up collections of source files to be built into an executable. This problem
>> crops up in at least two different flavors.
>>
>>
>> ** Lists of sources **
>>
>> The first, simplest version is that our project is quite large and has
>> many, many source files. It would be very impractical to try to put them
>> all in a huge list to supply to a Program builder, and this would also
>> architect out modularity of our build system, where source files would no
>> longer be handled locally and would instead have to be handled all in one
>> central clearing house. This would also break a mechanism we have which
>> lets users add new collections of source to the project to add their own
>> components. An approximate analogy would be kernel modules. It would not be
>> possible to add these extra directories without modifying the main projects
>> source if you had to, by definition, modify the huge list of source files
>> to extend the code base.
>>
>> To solve this problem, we currently declare python objects which
>> represent the source files, but don't actually mean anything to SCons
>> itself. These objects also carry "tags" so that they can say they should
>> only be included if the project includes its built in python interpreter,
>> or for certain unit tests, or if it's being built as a complete executable
>> instead of as a library.
>>
>> Then later, after all the SConscript files in subdirectories have be
>> processed, we have a mechanism to generate lists of source files with
>> and/or without various tags, and then we feed those into the actual
>> Program, SharedLibrary, or StaticLibrary builders.
>>
>> I would really rather have something more like this:
>>
>> Program('foo.bin', '${SOURCES.with_any_tags("main", "lib", "python")')
>>
>> and have that construction variable be expanded *after* all the sources
>> have been collected. It's relatively easy to centralize the declaration of
>> our central binary or libraries, but it's not really possible for things
>> like unit tests which are scattered throughout the code base (near what
>> they test) and can even come in through user additions.
>>
>>
>> ** Dependencies based on build products **
>>
>> Another more complicated problem is that our project is a simulator, and
>> the objects in the simulator are described using python. Our build actually
>> imports these modules, and then checks to see which different classes of
>> simulation objects have been set up. Then each of these are used to
>> generate additional c++ files which act as the glue between those python
>> classes and the c++ classes that underlie them.
>>
>> The problem here is similar, in that we need to collect all the python
>> modules, import them and generate a list of simulation objects, and then
>> based on that generate a collection of .cc files which will be built into
>> the simulator. We need to collect the .py files which is like above, but
>> then beyond that we need to run a build step, and then based on what
>> happens there add some number of .cc files to the build.
>>
>> What we're doing now is that we just have a step in the SConscript which
>> does all that in line, and then we set up additional source file
>> representing objects like I described above. This adds a decent amount of
>> complex custom code to our build scripts, and another sequential element to
>> the build process.
>>
>> Is there a way I can run the build step of importing all these .py, and
>> then add the extra .cc files to the build? I wasn't able to think of any
>> way to do that given how scons works, and this Stack Overflow post was the
>> closest I've found from anybody else, although it looks like it's abusing
>> internal interfaces and I'm pretty reluctant to do anything like that in
>> our production code:
>>
>>
>> https://stackoverflow.com/questions/24671859/scons-how-to-generate-dependencies-after-some-targets-have-been-built
>>
>> Are scanners the right way to do this somehow? Does SCons have any sort
>> of mechanism where it can re-scan nodes that have changed since the build
>> started?
>>
>>
>> ** Multiple invocations? **
>>
>> Another big hammer approach I'm considering is to write some sort of
>> wrapper script which will just invoke SCons multiple times, once for each
>> layer of dependencies in the build, and have it record the intermediate
>> results someplace for it rediscover between runs.
>>
>> This feels pretty clunky to me, and like something SCons should be
>> handling for me. Is there a better way?
>>
>> Gabe
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210726/5b386452/attachment.htm>