[Scons-users] persistent but local tweaks to a node's environment

[Gabe Black]

Roughly, but not quite. Some files need different CCFLAGS, but OBJSUFFIX is
on a per build flavor basis (.o for optimized, .do for debug, etc)

For #1, we have historically declared classes (Source, PySource, etc) which
individual SConscripts in subdirectories would declare which would be how
source files would be added to the build. For example, "Source('match.cc',
add_tags='gem5 trace')", Those would keep extra information in them like
additional settings for the environment, what categories the source should
belong to, etc. There were (are) also classes like this for top level
targets, which are declared either with explicit individual sources, or
combinations of tags which identify categories of sources, for instance
"GTest('trace.test', 'trace.test.cc', with_tag('gem5 trace'))". We would
then go through all the top level target objects and ask them to customize
the given environment for targets of their type (add the unit test
framework library, etc), and then to visit each instance of that class and
use that environment, plus any per instance tweaks, to define a rule to
build that target. Here, we would also evaluate the tag predicates as
applied to all declared source files to figure out what sources each target
would need. If a source was called for, we would visit it's side channel
class instance (Source, PySource, etc), and declare it as a StaticObject,
etc, with potentially it's own tweaks on top of all the other tweaks. For
that last step in particular, and when making the other tweaks in general,
we would env.Clone(), adjust the clone, and then declare the StaticObject,
etc, in that cloned environment.

Now what we do is slightly different. We still do most of the above, but
based on this email thread, I have removed the env.Clone() from individual
sources and instead ** a dict with extra settings to tweak the environment
for individual sources. Those now provide their tweaks in the form
discussed above, ie {"CCFLAGS": "${CCFLAGS} --extra-option"}. This has
removed some complexity and pulled away some of our extra code on top of
SCons, but as you can see there's a lot further to go. I'm hoping that my
suggestion above will help to further burn away these extra layers and
bookkeeping side channels so that our build relies more on SCons and less
on machinery in our scripts.

As far as variant directories, yes we use those. We actually use them at a
different level/dimension of variability in our build though. Roughly
speaking, we have a top level build directory which has all the
configuration invariant stuff, like local versions of external libraries,
etc. Then we have a collection of variant directories which have
configuration dependent stuff, where the name of each more or less
corresponds to the configuration they're using, X86 for a simulator which
targets x86, ARM_MOESI_hammer for an ARM simulator which uses the MOESI
coherence protocol, etc. Then within *these* directories, you can build
gem5.opt, gem5.fast, etc. These are also basically variants, but at that
level we don't use variant directories.

That also touches on another thing I would like to move away from in our
build scripts, which is that when setting up each variant directory, we
reprocess all of the SConscripts in the whole project again, but with the
config for that variant visible, ie TARGET_ISA would evaluate to 'x86'.
Usually people only build one configuration at a time, but when building
all the predefined configurations for testing, that's a lot of repeated
work.

My ideal at least for this pass would be that we go through all our
SConscripts once, set up everything once with SCons so it knows all the
rules and requirements for everything, and then at the end just say "go"
and have it work everything out with the rules it has. I would like to
reduce the number of things we iterate over to perhaps just the
environments for each build flavor, visiting each to tell scons "hang all
your rules off this too".

I am also thinking about ways to restructure the build directory itself. I
think having a single copy of config invariant build products is useful so
we don't end up building 12 identical copies of libelf, but having multiple
tiers of hierarchy is complex and makes it harder to understand the build
and how to get what you want out of it. I'm in the middle of a very long
term refactor of the project which is dissolving one major axis of
variability, the TARGET_ISA, but unfortunately there will still be one or
two left which will make it hard to retire the configuration dependent
variant directories for now. I would consider that a longer term goal,
where you could build a big, all inclusive version of the simulator if you
wanted to build and test all the features, rather than having to build all
the splinters which leads to extra copies of libelf, yatta yatta. Or in
other words, I want the configuration to turn on and off independent
features where possible, and not select from mutually exclusive features.

Also yes, we use CPPDEFINES for -Dxyz=1 type options, in my toy example it
was just a visible but also effectively inert flag to look for in command
lines.

Gabe

On Sat, Jul 31, 2021 at 4:54 PM Bill Deegan <bill at baddogconsulting.com>
wrote:

> Your solution sounds unnecessarily overcomplicated.
>
> Let me see if I understand what you're trying to do. (It's been a few days
> since I last engaged in your issue).
> 1.  Some target files require individually different CCFLAGS (in addition
> to a base set of flags) and/or OBJSUFFIX.
> 2. You build various flavors (debug, optimized,etc)
>
> For #1, how do you keep track of what flags/OBJSUFFIX each file requires?
> Also are there any other SCons variables which need to vary per file?
>
> For #2, are you using variant dirs? (If not, why?)
>
> BTW. You should not use CCFLAGS for -Dxyz=1, that should be in CPPDEFINES.
>
>
>
> On Fri, Jul 30, 2021 at 5:59 PM Gabe Black <gabe.black at gmail.com> wrote:
>
>> Hello again! I've had a chance to try implementing the idea in this toy
>> example into our actual build scripts, and while it got me closer, I'm
>> still running into problems.
>>
>> The way our scripts are structured is something roughly like this:
>>
>> main script:
>> find SConscripts in subdirectories and parse them
>>
>>     sub script(s):
>>     declare sources, mark some as part of special categories, or needing
>> environment tweaks,
>>     declare some unit tests which may include sources (by category) that
>> haven't been declared yet
>>
>> after processing subdirectories...
>> build environments for debug, optimized, profiled, etc, builds
>> in each of those environments, set up all top level targets (main
>> binaries, unit tests, library version of project, etc)
>>
>>
>> To set up each of those build environments, we use Clone(), and then
>> customize them with different flags, different object file or binary
>> suffixes, etc. Unfortunately it looks like this Clone()ing step breaks the
>> mechanism that worked in the toy example.
>>
>> A possible solution I've considered is that we might want to use
>> Override() and not Clone(), because as somebody mentioned, that creates an
>> overlay and does not completely copy and detach from the underlying
>> environment. This is not ideal given our current structure, since
>> Override() is not really a public mechanism, and if something down the line
>> building up an environment either globally, or (for instance) to use for
>> each of the unit tests tries to use Clone() itself, things will blow up.
>>
>> So then I thought about how I might use Override(), but without reaching
>> down into SCons's guts myself and using it directly.
>>
>> My thought thus far is that a builder can call other builders implicitly,
>> if it needs to build one of its sources, and it was set up with that other
>> builder as a source builder. Would it work to have, say, a pseudo top level
>> builder which builds some generic, internal only node (an Alias with a
>> weird name maybe), and then for each of its sources to imply a sub builder?
>> Then that sub builder would (I think) inherit the environment from the top
>> level builder, including any overrides. Those sources could be another
>> internal only node like "unit tests" or "libraries" or "main binaries", and
>> then each of those could in turn customize the environment further and get
>> down to the actual "Program" or "StaticLibrary" or ... builder which
>> actually builds an output.
>>
>> fast_environment.EveryThingBuilder(fast_override1="$BLAH $BLAH",
>> fast_override2="$FOO $BAR")
>> opt_environment.EveryThingBuilder(opt_override1=...)
>> debug_environment.EverythingBuilder(debug_override1=...)
>>
>> Importantly, this imaginary top level target would never be something the
>> user would actually build. It would just teach scons about a bunch of
>> targets by implying them as sources (would this actually work?)
>>
>> Another nice benefit of this approach is that the source files could be
>> built up in some sort of construction variable as the SConscripts are read
>> in, and since these are not actually consumed until all these implicit
>> builders get called, we wouldn't have to worry about the case where a
>> binary says they need all sources having to do with the built in python
>> interpreter, but those haven't all be declared yet.
>>
>> I feel like there's a solution in there, but I'm not sure if all my
>> assumptions are sound, and I've gotten a bit lost in the details. Can
>> someone please confirm whether this is something that even *could* work,
>> and if so help me rough out what pieces to use?
>>
>> Thanks so much for your help!
>>
>> Gabe
>>
>> On Sun, Jul 25, 2021 at 8:05 AM Gabe Black <gabe.black at gmail.com> wrote:
>>
>>> Sure. Here is an example SConstruct I was playing with:
>>>
>>> '''
>>> env = Environment()
>>>
>>> foo_o = env.Object('foo.c', CCFLAGS='${CCFLAGS} -DFOO=foo')
>>>
>>> env.Append(CCFLAGS=['-DBAR=bar'])
>>> foo = env.Program('foo', foo_o)
>>>
>>> Default(foo)
>>> '''
>>>
>>> When that builds foo.o, it always uses -DFOO=foo, and never -DFOO=bar,
>>> so the env.Append line seems to be clobbered by the override in
>>> env.Object().
>>>
>>> Thinking about what you said though, I tried a little experiment which
>>> seemed to work more like what I would expect
>>>
>>> '''
>>> env = Environment()
>>>
>>> env.Append(CCFLAGS=['${CCFLAGS_extra}'])
>>> foo_o = env.Object('foo.c', CCFLAGS_extra='-DFOO=foo')
>>>
>>> env.Append(CCFLAGS=['-DBAR=bar'])
>>> foo = env.Program('foo', foo_o)
>>>
>>> Default(foo)
>>> '''
>>>
>>> With that I get both -DFOO=bar and -DFOO=foo. I can probably work with
>>> that since everything I'd need to add is local to env.Object, and I might
>>> even be able to wrap it in something to make it a little less cumbersome.
>>>
>>> Related to this, I know you can override a variable using FOO='bar' in a
>>> builder, but is there a way to Append? I usually want to, for instance, add
>>> extra compiler flags without clobbering everything that's already there.
>>> That's what I was *hoping* to achieve with the CCFLAGS='${CCFLAGS} ...',
>>> but I don't think that's what it ended up doing!
>>>
>>> Gabe
>>>
>>> On Sun, Jul 25, 2021 at 6:52 AM Mats Wichmann <mats at wichmann.us> wrote:
>>>
>>>> On 7/24/21 4:18 AM, Gabe Black wrote:
>>>> > Hi! The project I work on builds several different versions of its
>>>> > outputs, like a debug version, an optimized version, etc. Also, some
>>>> > files in the project, for instance automatically generated files,
>>>> need
>>>> > to have slightly customized build flags, like disabling certain
>>>> warnings
>>>> > that are false positives and/or are in source we can't change.
>>>> >
>>>> > Right now we handle that by creating an environment for each build,
>>>> > which I think is the standard approach. Unfortunately, we then have
>>>> to
>>>> > have a separate step which goes through and generates new
>>>> environments
>>>> > based on those for each file that needs its slightly customized build
>>>> flags.
>>>> >
>>>> > What I would *like* to do, is to be able to specify for any given
>>>> node
>>>> > that it should build using the flags that would be implied by the
>>>> > environment that's pulling it in, except that they should be tweaked
>>>> > just a little bit. Something like:
>>>> >
>>>> > Object('foo.cc', '-Wno-deprecated-copy')
>>>> >
>>>> > and then:
>>>> >
>>>> > opt.Program('foo.opt', 'foo.cc', CCFLAGS='${CCFLAGS} -O3')
>>>> > debug.Program('foo.debug', 'foo.cc', CCFLAGS='${CCFLAGS} -g',
>>>> > OBJSUFFIX='.do')
>>>> > etc
>>>> >
>>>> > and then have foo.do build with "-Wno-deprecated-copy -g", and foo.o
>>>> > build with "-Wno-deprecated-copy -O3"
>>>> >
>>>> > The problem is, that as soon as I tell SCons to use custom variables,
>>>> it
>>>> > Clone-s the underlying environment and disassociates from the
>>>> underlying
>>>> > environment.
>>>>
>>>> could you explain what you're seeing a little further?
>>>>
>>>> In the case of construction variable overrides in a builder call, it
>>>> doesn't Clone, as that's considered expensive, it just builds an
>>>> override dictionary, and it doesn't disassociate - it retains the
>>>> underlying construction environment and only applies the override dict
>>>> if there were any overrides - so it's supposed to be a kind of
>>>> copy-on-write setup.  Since this doesn't seem to match what you're
>>>> seeing it would be good to figure out what's actually happening.
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210731/3a4e1dc0/attachment-0001.htm>