[Scons-users] persistent but local tweaks to a node's environment

[Bill Deegan]

So at the time you call StaticObject(), for example, do you know what flags
are needed for that  source?

(if possible shorter paragraphs would make your explanations easier to
digest..)

Also according to your response above, I would set OBJSUFFIX in the
Environment() which should be per build flavor.

re a single pass in the SConscripts, if you're not directly calling SCons
methods, but rather building up a "database" of the build which then gets
processed, then it'd be possible to avoid multiple SConscript calls to each
SConscript, however if the logic in those SConscripts doesn't different
things for different variants, it seems like you'd only be adding more
complexity to your "database" of build info.

Honestly, the further you get from plain SConscripts, the harder it will be
for developers to learn and work on your build system.






On Sat, Jul 31, 2021 at 10:44 PM Gabe Black <gabe.black at gmail.com> wrote:

> Roughly, but not quite. Some files need different CCFLAGS, but
> OBJSUFFIX is on a per build flavor basis (.o for optimized, .do for debug,
> etc)
>
> For #1, we have historically declared classes (Source, PySource, etc)
> which individual SConscripts in subdirectories would declare which would be
> how source files would be added to the build. For example,
> "Source('match.cc', add_tags='gem5 trace')", Those would keep extra
> information in them like additional settings for the environment, what
> categories the source should belong to, etc. There were (are) also classes
> like this for top level targets, which are declared either with explicit
> individual sources, or combinations of tags which identify categories of
> sources, for instance "GTest('trace.test', 'trace.test.cc',
> with_tag('gem5 trace'))". We would then go through all the top level target
> objects and ask them to customize the given environment for targets of
> their type (add the unit test framework library, etc), and then to visit
> each instance of that class and use that environment, plus any per instance
> tweaks, to define a rule to build that target. Here, we would also evaluate
> the tag predicates as applied to all declared source files to figure out
> what sources each target would need. If a source was called for, we would
> visit it's side channel class instance (Source, PySource, etc), and declare
> it as a StaticObject, etc, with potentially it's own tweaks on top of all
> the other tweaks. For that last step in particular, and when making the
> other tweaks in general, we would env.Clone(), adjust the clone, and then
> declare the StaticObject, etc, in that cloned environment.
>
> Now what we do is slightly different. We still do most of the above, but
> based on this email thread, I have removed the env.Clone() from individual
> sources and instead ** a dict with extra settings to tweak the environment
> for individual sources. Those now provide their tweaks in the form
> discussed above, ie {"CCFLAGS": "${CCFLAGS} --extra-option"}. This has
> removed some complexity and pulled away some of our extra code on top of
> SCons, but as you can see there's a lot further to go. I'm hoping that my
> suggestion above will help to further burn away these extra layers and
> bookkeeping side channels so that our build relies more on SCons and less
> on machinery in our scripts.
>
> As far as variant directories, yes we use those. We actually use them at a
> different level/dimension of variability in our build though. Roughly
> speaking, we have a top level build directory which has all the
> configuration invariant stuff, like local versions of external libraries,
> etc. Then we have a collection of variant directories which have
> configuration dependent stuff, where the name of each more or less
> corresponds to the configuration they're using, X86 for a simulator which
> targets x86, ARM_MOESI_hammer for an ARM simulator which uses the MOESI
> coherence protocol, etc. Then within *these* directories, you can build
> gem5.opt, gem5.fast, etc. These are also basically variants, but at that
> level we don't use variant directories.
>
> That also touches on another thing I would like to move away from in our
> build scripts, which is that when setting up each variant directory, we
> reprocess all of the SConscripts in the whole project again, but with the
> config for that variant visible, ie TARGET_ISA would evaluate to 'x86'.
> Usually people only build one configuration at a time, but when building
> all the predefined configurations for testing, that's a lot of repeated
> work.
>
> My ideal at least for this pass would be that we go through all our
> SConscripts once, set up everything once with SCons so it knows all the
> rules and requirements for everything, and then at the end just say "go"
> and have it work everything out with the rules it has. I would like to
> reduce the number of things we iterate over to perhaps just the
> environments for each build flavor, visiting each to tell scons "hang all
> your rules off this too".
>
> I am also thinking about ways to restructure the build directory itself. I
> think having a single copy of config invariant build products is useful so
> we don't end up building 12 identical copies of libelf, but having multiple
> tiers of hierarchy is complex and makes it harder to understand the build
> and how to get what you want out of it. I'm in the middle of a very long
> term refactor of the project which is dissolving one major axis of
> variability, the TARGET_ISA, but unfortunately there will still be one or
> two left which will make it hard to retire the configuration dependent
> variant directories for now. I would consider that a longer term goal,
> where you could build a big, all inclusive version of the simulator if you
> wanted to build and test all the features, rather than having to build all
> the splinters which leads to extra copies of libelf, yatta yatta. Or in
> other words, I want the configuration to turn on and off independent
> features where possible, and not select from mutually exclusive features.
>
> Also yes, we use CPPDEFINES for -Dxyz=1 type options, in my toy example it
> was just a visible but also effectively inert flag to look for in command
> lines.
>
> Gabe
>
> On Sat, Jul 31, 2021 at 4:54 PM Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> Your solution sounds unnecessarily overcomplicated.
>>
>> Let me see if I understand what you're trying to do. (It's been a few
>> days since I last engaged in your issue).
>> 1.  Some target files require individually different CCFLAGS (in addition
>> to a base set of flags) and/or OBJSUFFIX.
>> 2. You build various flavors (debug, optimized,etc)
>>
>> For #1, how do you keep track of what flags/OBJSUFFIX each file requires?
>> Also are there any other SCons variables which need to vary per file?
>>
>> For #2, are you using variant dirs? (If not, why?)
>>
>> BTW. You should not use CCFLAGS for -Dxyz=1, that should be in CPPDEFINES.
>>
>>
>>
>> On Fri, Jul 30, 2021 at 5:59 PM Gabe Black <gabe.black at gmail.com> wrote:
>>
>>> Hello again! I've had a chance to try implementing the idea in this toy
>>> example into our actual build scripts, and while it got me closer, I'm
>>> still running into problems.
>>>
>>> The way our scripts are structured is something roughly like this:
>>>
>>> main script:
>>> find SConscripts in subdirectories and parse them
>>>
>>>     sub script(s):
>>>     declare sources, mark some as part of special categories, or needing
>>> environment tweaks,
>>>     declare some unit tests which may include sources (by category) that
>>> haven't been declared yet
>>>
>>> after processing subdirectories...
>>> build environments for debug, optimized, profiled, etc, builds
>>> in each of those environments, set up all top level targets (main
>>> binaries, unit tests, library version of project, etc)
>>>
>>>
>>> To set up each of those build environments, we use Clone(), and then
>>> customize them with different flags, different object file or binary
>>> suffixes, etc. Unfortunately it looks like this Clone()ing step breaks the
>>> mechanism that worked in the toy example.
>>>
>>> A possible solution I've considered is that we might want to use
>>> Override() and not Clone(), because as somebody mentioned, that creates an
>>> overlay and does not completely copy and detach from the underlying
>>> environment. This is not ideal given our current structure, since
>>> Override() is not really a public mechanism, and if something down the line
>>> building up an environment either globally, or (for instance) to use for
>>> each of the unit tests tries to use Clone() itself, things will blow up.
>>>
>>> So then I thought about how I might use Override(), but without reaching
>>> down into SCons's guts myself and using it directly.
>>>
>>> My thought thus far is that a builder can call other builders
>>> implicitly, if it needs to build one of its sources, and it was set up with
>>> that other builder as a source builder. Would it work to have, say, a
>>> pseudo top level builder which builds some generic, internal only node (an
>>> Alias with a weird name maybe), and then for each of its sources to imply a
>>> sub builder? Then that sub builder would (I think) inherit the environment
>>> from the top level builder, including any overrides. Those sources could be
>>> another internal only node like "unit tests" or "libraries" or "main
>>> binaries", and then each of those could in turn customize the environment
>>> further and get down to the actual "Program" or "StaticLibrary" or ...
>>> builder which actually builds an output.
>>>
>>> fast_environment.EveryThingBuilder(fast_override1="$BLAH $BLAH",
>>> fast_override2="$FOO $BAR")
>>> opt_environment.EveryThingBuilder(opt_override1=...)
>>> debug_environment.EverythingBuilder(debug_override1=...)
>>>
>>> Importantly, this imaginary top level target would never be something
>>> the user would actually build. It would just teach scons about a bunch of
>>> targets by implying them as sources (would this actually work?)
>>>
>>> Another nice benefit of this approach is that the source files could be
>>> built up in some sort of construction variable as the SConscripts are read
>>> in, and since these are not actually consumed until all these implicit
>>> builders get called, we wouldn't have to worry about the case where a
>>> binary says they need all sources having to do with the built in python
>>> interpreter, but those haven't all be declared yet.
>>>
>>> I feel like there's a solution in there, but I'm not sure if all my
>>> assumptions are sound, and I've gotten a bit lost in the details. Can
>>> someone please confirm whether this is something that even *could* work,
>>> and if so help me rough out what pieces to use?
>>>
>>> Thanks so much for your help!
>>>
>>> Gabe
>>>
>>> On Sun, Jul 25, 2021 at 8:05 AM Gabe Black <gabe.black at gmail.com> wrote:
>>>
>>>> Sure. Here is an example SConstruct I was playing with:
>>>>
>>>> '''
>>>> env = Environment()
>>>>
>>>> foo_o = env.Object('foo.c', CCFLAGS='${CCFLAGS} -DFOO=foo')
>>>>
>>>> env.Append(CCFLAGS=['-DBAR=bar'])
>>>> foo = env.Program('foo', foo_o)
>>>>
>>>> Default(foo)
>>>> '''
>>>>
>>>> When that builds foo.o, it always uses -DFOO=foo, and never -DFOO=bar,
>>>> so the env.Append line seems to be clobbered by the override in
>>>> env.Object().
>>>>
>>>> Thinking about what you said though, I tried a little experiment which
>>>> seemed to work more like what I would expect
>>>>
>>>> '''
>>>> env = Environment()
>>>>
>>>> env.Append(CCFLAGS=['${CCFLAGS_extra}'])
>>>> foo_o = env.Object('foo.c', CCFLAGS_extra='-DFOO=foo')
>>>>
>>>> env.Append(CCFLAGS=['-DBAR=bar'])
>>>> foo = env.Program('foo', foo_o)
>>>>
>>>> Default(foo)
>>>> '''
>>>>
>>>> With that I get both -DFOO=bar and -DFOO=foo. I can probably work with
>>>> that since everything I'd need to add is local to env.Object, and I might
>>>> even be able to wrap it in something to make it a little less cumbersome.
>>>>
>>>> Related to this, I know you can override a variable using FOO='bar' in
>>>> a builder, but is there a way to Append? I usually want to, for instance,
>>>> add extra compiler flags without clobbering everything that's already
>>>> there. That's what I was *hoping* to achieve with the CCFLAGS='${CCFLAGS}
>>>> ...', but I don't think that's what it ended up doing!
>>>>
>>>> Gabe
>>>>
>>>> On Sun, Jul 25, 2021 at 6:52 AM Mats Wichmann <mats at wichmann.us> wrote:
>>>>
>>>>> On 7/24/21 4:18 AM, Gabe Black wrote:
>>>>> > Hi! The project I work on builds several different versions of its
>>>>> > outputs, like a debug version, an optimized version, etc. Also, some
>>>>> > files in the project, for instance automatically generated files,
>>>>> need
>>>>> > to have slightly customized build flags, like disabling certain
>>>>> warnings
>>>>> > that are false positives and/or are in source we can't change.
>>>>> >
>>>>> > Right now we handle that by creating an environment for each build,
>>>>> > which I think is the standard approach. Unfortunately, we then have
>>>>> to
>>>>> > have a separate step which goes through and generates new
>>>>> environments
>>>>> > based on those for each file that needs its slightly customized
>>>>> build flags.
>>>>> >
>>>>> > What I would *like* to do, is to be able to specify for any given
>>>>> node
>>>>> > that it should build using the flags that would be implied by the
>>>>> > environment that's pulling it in, except that they should be tweaked
>>>>> > just a little bit. Something like:
>>>>> >
>>>>> > Object('foo.cc', '-Wno-deprecated-copy')
>>>>> >
>>>>> > and then:
>>>>> >
>>>>> > opt.Program('foo.opt', 'foo.cc', CCFLAGS='${CCFLAGS} -O3')
>>>>> > debug.Program('foo.debug', 'foo.cc', CCFLAGS='${CCFLAGS} -g',
>>>>> > OBJSUFFIX='.do')
>>>>> > etc
>>>>> >
>>>>> > and then have foo.do build with "-Wno-deprecated-copy -g", and foo.o
>>>>> > build with "-Wno-deprecated-copy -O3"
>>>>> >
>>>>> > The problem is, that as soon as I tell SCons to use custom
>>>>> variables, it
>>>>> > Clone-s the underlying environment and disassociates from the
>>>>> underlying
>>>>> > environment.
>>>>>
>>>>> could you explain what you're seeing a little further?
>>>>>
>>>>> In the case of construction variable overrides in a builder call, it
>>>>> doesn't Clone, as that's considered expensive, it just builds an
>>>>> override dictionary, and it doesn't disassociate - it retains the
>>>>> underlying construction environment and only applies the override dict
>>>>> if there were any overrides - so it's supposed to be a kind of
>>>>> copy-on-write setup.  Since this doesn't seem to match what you're
>>>>> seeing it would be good to figure out what's actually happening.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210801/12cf8d76/attachment-0001.htm>