[Scons-users] collecting groups of files as sources

Mon Jul 26 19:13:34 EDT 2021

Agreed.

Gabe

On Mon, Jul 26, 2021 at 3:41 PM Bill Deegan <bill at baddogconsulting.com>
wrote:

> Gabe,
>
> As a general rule, I advise against putting too much logic in python
> actions in SCons.
> In general if it's anything heavyweight, you're better off with a script
> that scons runs, rather that a big blob of logic in a python action
> function.
>
> -Bill
>
> On Mon, Jul 26, 2021 at 3:30 PM Gabe Black <gabe.black at gmail.com> wrote:
>
>> Thanks! Sure, I'll join your discord server! Please, take your time, your
>> suggestions are of course welcome. I'm in the midst of a deep
>> clean/refactoring of our build process, and you can find my pending changes
>> in this series in our code review system:
>>
>> https://gem5-review.googlesource.com/c/public/gem5/+/48381
>>
>> My current goal is to reduce the amount of custom code and extra
>> machinery we have, and push the work into SCons itself. In addition to
>> making things simpler in general, it will hopefully also make it easier to
>> work on since people will just need to know how SCons works, not how SCons
>> works and how our extra layers work on top of it.
>>
>> Gabe
>>
>> On Sun, Jul 25, 2021 at 8:35 PM Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>>> Gabe,
>>> Welcome!
>>>
>>> Gem5 is a long time SCons user.
>>> I've helped a few of your users over the past few years as they ran into
>>> various issue.
>>> Any chance we can get you to join our discord server?
>>> https://discord.gg/pejaFYrD9n
>>>
>>> I've just cloned your repo to take a look.
>>> I'm on vacation for a few days, but if you give me a bit of time, I'll
>>> be able to make better suggestions.
>>>
>>> On possibility comes to mind, any chance that you could segregate the
>>> various targets into separate directories on a per program basis?
>>> That would simplify to Glob('#/build/program1/objects/*.obj') for
>>> example.
>>> (I'm guessing it wouldn't be this simple, but worth asking)
>>>
>>> -Bill
>>> SCons Project Co-Manager
>>>
>>> On Sat, Jul 24, 2021 at 3:02 AM Gabe Black <gabe.black at gmail.com> wrote:
>>>
>>>> Hi, I'm a core developer on large project which uses SCons as our build
>>>> system (gem5.org), and I'm in the process of refactoring our build so
>>>> that it has less custom code and defers more work to SCons itself.
>>>>
>>>> The biggest sticking points so far have generally been around gathering
>>>> up collections of source files to be built into an executable. This problem
>>>> crops up in at least two different flavors.
>>>>
>>>>
>>>> ** Lists of sources **
>>>>
>>>> The first, simplest version is that our project is quite large and has
>>>> many, many source files. It would be very impractical to try to put them
>>>> all in a huge list to supply to a Program builder, and this would also
>>>> architect out modularity of our build system, where source files would no
>>>> longer be handled locally and would instead have to be handled all in one
>>>> central clearing house. This would also break a mechanism we have which
>>>> lets users add new collections of source to the project to add their own
>>>> components. An approximate analogy would be kernel modules. It would not be
>>>> possible to add these extra directories without modifying the main projects
>>>> source if you had to, by definition, modify the huge list of source files
>>>> to extend the code base.
>>>>
>>>> To solve this problem, we currently declare python objects which
>>>> represent the source files, but don't actually mean anything to SCons
>>>> itself. These objects also carry "tags" so that they can say they should
>>>> only be included if the project includes its built in python interpreter,
>>>> or for certain unit tests, or if it's being built as a complete executable
>>>> instead of as a library.
>>>>
>>>> Then later, after all the SConscript files in subdirectories have be
>>>> processed, we have a mechanism to generate lists of source files with
>>>> and/or without various tags, and then we feed those into the actual
>>>> Program, SharedLibrary, or StaticLibrary builders.
>>>>
>>>> I would really rather have something more like this:
>>>>
>>>> Program('foo.bin', '${SOURCES.with_any_tags("main", "lib", "python")')
>>>>
>>>> and have that construction variable be expanded *after* all the sources
>>>> have been collected. It's relatively easy to centralize the declaration of
>>>> our central binary or libraries, but it's not really possible for things
>>>> like unit tests which are scattered throughout the code base (near what
>>>> they test) and can even come in through user additions.
>>>>
>>>>
>>>> ** Dependencies based on build products **
>>>>
>>>> Another more complicated problem is that our project is a simulator,
>>>> and the objects in the simulator are described using python. Our build
>>>> actually imports these modules, and then checks to see which different
>>>> classes of simulation objects have been set up. Then each of these are used
>>>> to generate additional c++ files which act as the glue between those python
>>>> classes and the c++ classes that underlie them.
>>>>
>>>> The problem here is similar, in that we need to collect all the python
>>>> modules, import them and generate a list of simulation objects, and then
>>>> based on that generate a collection of .cc files which will be built into
>>>> the simulator. We need to collect the .py files which is like above, but
>>>> then beyond that we need to run a build step, and then based on what
>>>> happens there add some number of .cc files to the build.
>>>>
>>>> What we're doing now is that we just have a step in the SConscript
>>>> which does all that in line, and then we set up additional source file
>>>> representing objects like I described above. This adds a decent amount of
>>>> complex custom code to our build scripts, and another sequential element to
>>>> the build process.
>>>>
>>>> Is there a way I can run the build step of importing all these .py, and
>>>> then add the extra .cc files to the build? I wasn't able to think of any
>>>> way to do that given how scons works, and this Stack Overflow post was the
>>>> closest I've found from anybody else, although it looks like it's abusing
>>>> internal interfaces and I'm pretty reluctant to do anything like that in
>>>> our production code:
>>>>
>>>>
>>>> https://stackoverflow.com/questions/24671859/scons-how-to-generate-dependencies-after-some-targets-have-been-built
>>>>
>>>> Are scanners the right way to do this somehow? Does SCons have any sort
>>>> of mechanism where it can re-scan nodes that have changed since the build
>>>> started?
>>>>
>>>>
>>>> ** Multiple invocations? **
>>>>
>>>> Another big hammer approach I'm considering is to write some sort of
>>>> wrapper script which will just invoke SCons multiple times, once for each
>>>> layer of dependencies in the build, and have it record the intermediate
>>>> results someplace for it rediscover between runs.
>>>>
>>>> This feels pretty clunky to me, and like something SCons should be
>>>> handling for me. Is there a better way?
>>>>
>>>> Gabe
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210726/d606f897/attachment-0001.htm>