[Scons-users] collecting groups of files as sources

[Bill Deegan]

Gabe,

As a general rule, I advise against putting too much logic in python
actions in SCons.
In general if it's anything heavyweight, you're better off with a script
that scons runs, rather that a big blob of logic in a python action
function.

-Bill

On Mon, Jul 26, 2021 at 3:30 PM Gabe Black <gabe.black at gmail.com> wrote:

> Thanks! Sure, I'll join your discord server! Please, take your time, your
> suggestions are of course welcome. I'm in the midst of a deep
> clean/refactoring of our build process, and you can find my pending changes
> in this series in our code review system:
>
> https://gem5-review.googlesource.com/c/public/gem5/+/48381
>
> My current goal is to reduce the amount of custom code and extra machinery
> we have, and push the work into SCons itself. In addition to making things
> simpler in general, it will hopefully also make it easier to work on since
> people will just need to know how SCons works, not how SCons works and how
> our extra layers work on top of it.
>
> Gabe
>
> On Sun, Jul 25, 2021 at 8:35 PM Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> Gabe,
>> Welcome!
>>
>> Gem5 is a long time SCons user.
>> I've helped a few of your users over the past few years as they ran into
>> various issue.
>> Any chance we can get you to join our discord server?
>> https://discord.gg/pejaFYrD9n
>>
>> I've just cloned your repo to take a look.
>> I'm on vacation for a few days, but if you give me a bit of time, I'll be
>> able to make better suggestions.
>>
>> On possibility comes to mind, any chance that you could segregate the
>> various targets into separate directories on a per program basis?
>> That would simplify to Glob('#/build/program1/objects/*.obj') for example.
>> (I'm guessing it wouldn't be this simple, but worth asking)
>>
>> -Bill
>> SCons Project Co-Manager
>>
>> On Sat, Jul 24, 2021 at 3:02 AM Gabe Black <gabe.black at gmail.com> wrote:
>>
>>> Hi, I'm a core developer on large project which uses SCons as our build
>>> system (gem5.org), and I'm in the process of refactoring our build so
>>> that it has less custom code and defers more work to SCons itself.
>>>
>>> The biggest sticking points so far have generally been around gathering
>>> up collections of source files to be built into an executable. This problem
>>> crops up in at least two different flavors.
>>>
>>>
>>> ** Lists of sources **
>>>
>>> The first, simplest version is that our project is quite large and has
>>> many, many source files. It would be very impractical to try to put them
>>> all in a huge list to supply to a Program builder, and this would also
>>> architect out modularity of our build system, where source files would no
>>> longer be handled locally and would instead have to be handled all in one
>>> central clearing house. This would also break a mechanism we have which
>>> lets users add new collections of source to the project to add their own
>>> components. An approximate analogy would be kernel modules. It would not be
>>> possible to add these extra directories without modifying the main projects
>>> source if you had to, by definition, modify the huge list of source files
>>> to extend the code base.
>>>
>>> To solve this problem, we currently declare python objects which
>>> represent the source files, but don't actually mean anything to SCons
>>> itself. These objects also carry "tags" so that they can say they should
>>> only be included if the project includes its built in python interpreter,
>>> or for certain unit tests, or if it's being built as a complete executable
>>> instead of as a library.
>>>
>>> Then later, after all the SConscript files in subdirectories have be
>>> processed, we have a mechanism to generate lists of source files with
>>> and/or without various tags, and then we feed those into the actual
>>> Program, SharedLibrary, or StaticLibrary builders.
>>>
>>> I would really rather have something more like this:
>>>
>>> Program('foo.bin', '${SOURCES.with_any_tags("main", "lib", "python")')
>>>
>>> and have that construction variable be expanded *after* all the sources
>>> have been collected. It's relatively easy to centralize the declaration of
>>> our central binary or libraries, but it's not really possible for things
>>> like unit tests which are scattered throughout the code base (near what
>>> they test) and can even come in through user additions.
>>>
>>>
>>> ** Dependencies based on build products **
>>>
>>> Another more complicated problem is that our project is a simulator, and
>>> the objects in the simulator are described using python. Our build actually
>>> imports these modules, and then checks to see which different classes of
>>> simulation objects have been set up. Then each of these are used to
>>> generate additional c++ files which act as the glue between those python
>>> classes and the c++ classes that underlie them.
>>>
>>> The problem here is similar, in that we need to collect all the python
>>> modules, import them and generate a list of simulation objects, and then
>>> based on that generate a collection of .cc files which will be built into
>>> the simulator. We need to collect the .py files which is like above, but
>>> then beyond that we need to run a build step, and then based on what
>>> happens there add some number of .cc files to the build.
>>>
>>> What we're doing now is that we just have a step in the SConscript which
>>> does all that in line, and then we set up additional source file
>>> representing objects like I described above. This adds a decent amount of
>>> complex custom code to our build scripts, and another sequential element to
>>> the build process.
>>>
>>> Is there a way I can run the build step of importing all these .py, and
>>> then add the extra .cc files to the build? I wasn't able to think of any
>>> way to do that given how scons works, and this Stack Overflow post was the
>>> closest I've found from anybody else, although it looks like it's abusing
>>> internal interfaces and I'm pretty reluctant to do anything like that in
>>> our production code:
>>>
>>>
>>> https://stackoverflow.com/questions/24671859/scons-how-to-generate-dependencies-after-some-targets-have-been-built
>>>
>>> Are scanners the right way to do this somehow? Does SCons have any sort
>>> of mechanism where it can re-scan nodes that have changed since the build
>>> started?
>>>
>>>
>>> ** Multiple invocations? **
>>>
>>> Another big hammer approach I'm considering is to write some sort of
>>> wrapper script which will just invoke SCons multiple times, once for each
>>> layer of dependencies in the build, and have it record the intermediate
>>> results someplace for it rediscover between runs.
>>>
>>> This feels pretty clunky to me, and like something SCons should be
>>> handling for me. Is there a better way?
>>>
>>> Gabe
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210726/12d499d1/attachment-0001.htm>