[Scons-users] persistent but local tweaks to a node's environment

[Gabe Black]

Hello again! I've had a chance to try implementing the idea in this toy
example into our actual build scripts, and while it got me closer, I'm
still running into problems.

The way our scripts are structured is something roughly like this:

main script:
find SConscripts in subdirectories and parse them

    sub script(s):
    declare sources, mark some as part of special categories, or needing
environment tweaks,
    declare some unit tests which may include sources (by category) that
haven't been declared yet

after processing subdirectories...
build environments for debug, optimized, profiled, etc, builds
in each of those environments, set up all top level targets (main binaries,
unit tests, library version of project, etc)


To set up each of those build environments, we use Clone(), and then
customize them with different flags, different object file or binary
suffixes, etc. Unfortunately it looks like this Clone()ing step breaks the
mechanism that worked in the toy example.

A possible solution I've considered is that we might want to use Override()
and not Clone(), because as somebody mentioned, that creates an overlay and
does not completely copy and detach from the underlying environment. This
is not ideal given our current structure, since Override() is not really a
public mechanism, and if something down the line building up an environment
either globally, or (for instance) to use for each of the unit tests tries
to use Clone() itself, things will blow up.

So then I thought about how I might use Override(), but without reaching
down into SCons's guts myself and using it directly.

My thought thus far is that a builder can call other builders implicitly,
if it needs to build one of its sources, and it was set up with that other
builder as a source builder. Would it work to have, say, a pseudo top level
builder which builds some generic, internal only node (an Alias with a
weird name maybe), and then for each of its sources to imply a sub builder?
Then that sub builder would (I think) inherit the environment from the top
level builder, including any overrides. Those sources could be another
internal only node like "unit tests" or "libraries" or "main binaries", and
then each of those could in turn customize the environment further and get
down to the actual "Program" or "StaticLibrary" or ... builder which
actually builds an output.

fast_environment.EveryThingBuilder(fast_override1="$BLAH $BLAH",
fast_override2="$FOO $BAR")
opt_environment.EveryThingBuilder(opt_override1=...)
debug_environment.EverythingBuilder(debug_override1=...)

Importantly, this imaginary top level target would never be something the
user would actually build. It would just teach scons about a bunch of
targets by implying them as sources (would this actually work?)

Another nice benefit of this approach is that the source files could be
built up in some sort of construction variable as the SConscripts are read
in, and since these are not actually consumed until all these implicit
builders get called, we wouldn't have to worry about the case where a
binary says they need all sources having to do with the built in python
interpreter, but those haven't all be declared yet.

I feel like there's a solution in there, but I'm not sure if all my
assumptions are sound, and I've gotten a bit lost in the details. Can
someone please confirm whether this is something that even *could* work,
and if so help me rough out what pieces to use?

Thanks so much for your help!

Gabe

On Sun, Jul 25, 2021 at 8:05 AM Gabe Black <gabe.black at gmail.com> wrote:

> Sure. Here is an example SConstruct I was playing with:
>
> '''
> env = Environment()
>
> foo_o = env.Object('foo.c', CCFLAGS='${CCFLAGS} -DFOO=foo')
>
> env.Append(CCFLAGS=['-DBAR=bar'])
> foo = env.Program('foo', foo_o)
>
> Default(foo)
> '''
>
> When that builds foo.o, it always uses -DFOO=foo, and never -DFOO=bar, so
> the env.Append line seems to be clobbered by the override in env.Object().
>
> Thinking about what you said though, I tried a little experiment which
> seemed to work more like what I would expect
>
> '''
> env = Environment()
>
> env.Append(CCFLAGS=['${CCFLAGS_extra}'])
> foo_o = env.Object('foo.c', CCFLAGS_extra='-DFOO=foo')
>
> env.Append(CCFLAGS=['-DBAR=bar'])
> foo = env.Program('foo', foo_o)
>
> Default(foo)
> '''
>
> With that I get both -DFOO=bar and -DFOO=foo. I can probably work with
> that since everything I'd need to add is local to env.Object, and I might
> even be able to wrap it in something to make it a little less cumbersome.
>
> Related to this, I know you can override a variable using FOO='bar' in a
> builder, but is there a way to Append? I usually want to, for instance, add
> extra compiler flags without clobbering everything that's already there.
> That's what I was *hoping* to achieve with the CCFLAGS='${CCFLAGS} ...',
> but I don't think that's what it ended up doing!
>
> Gabe
>
> On Sun, Jul 25, 2021 at 6:52 AM Mats Wichmann <mats at wichmann.us> wrote:
>
>> On 7/24/21 4:18 AM, Gabe Black wrote:
>> > Hi! The project I work on builds several different versions of its
>> > outputs, like a debug version, an optimized version, etc. Also, some
>> > files in the project, for instance automatically generated files, need
>> > to have slightly customized build flags, like disabling certain
>> warnings
>> > that are false positives and/or are in source we can't change.
>> >
>> > Right now we handle that by creating an environment for each build,
>> > which I think is the standard approach. Unfortunately, we then have to
>> > have a separate step which goes through and generates new environments
>> > based on those for each file that needs its slightly customized build
>> flags.
>> >
>> > What I would *like* to do, is to be able to specify for any given node
>> > that it should build using the flags that would be implied by the
>> > environment that's pulling it in, except that they should be tweaked
>> > just a little bit. Something like:
>> >
>> > Object('foo.cc', '-Wno-deprecated-copy')
>> >
>> > and then:
>> >
>> > opt.Program('foo.opt', 'foo.cc', CCFLAGS='${CCFLAGS} -O3')
>> > debug.Program('foo.debug', 'foo.cc', CCFLAGS='${CCFLAGS} -g',
>> > OBJSUFFIX='.do')
>> > etc
>> >
>> > and then have foo.do build with "-Wno-deprecated-copy -g", and foo.o
>> > build with "-Wno-deprecated-copy -O3"
>> >
>> > The problem is, that as soon as I tell SCons to use custom variables,
>> it
>> > Clone-s the underlying environment and disassociates from the
>> underlying
>> > environment.
>>
>> could you explain what you're seeing a little further?
>>
>> In the case of construction variable overrides in a builder call, it
>> doesn't Clone, as that's considered expensive, it just builds an
>> override dictionary, and it doesn't disassociate - it retains the
>> underlying construction environment and only applies the override dict
>> if there were any overrides - so it's supposed to be a kind of
>> copy-on-write setup.  Since this doesn't seem to match what you're
>> seeing it would be good to figure out what's actually happening.
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210730/14063221/attachment.htm>