[Scons-users] Help my build get faster... (was SCons and parts)

[Jason Kenny]

I would agree with Bill in that if you are compiling a lot of objects, this might be better.

If you use Parts I would say it is better for cases in when you have a notion of component ( ie component might be like a project in VS). Parts would define a component with a name and a version and some set of depends. Parts will at this point map exported outputs from one component to the inputs to dependent component for you and will load the set you need to build a given component target. Here Parts can has different load logics you can use to help define how Parts should load the component. The default logic should be the best, however this is a no depends options to load only the targets provided and assume the base components exists. This can lead to bad build states, but is useful in quick compiler iteration. Since Parts is managing the component loading it know what set of the tree to load from file or cache based on what you state as a target to build and the depends that are defined per component.

I you are building 2000 object files it might be better to do what Bill suggests and add the use of Alias() to a set of targets. Parts does this inherently per component, but it seems that you might have a few groups of items to build. In this case you can manually define the groups and assign those outputs to an Alias() such as :


for k, f in group1:
   t = env.MyCompiler(k, opts=f['opts'], includes=f['include'])
   group_1.append(t)
   all_targets.append(t)


for k, f in group2:
   t = env.MyCompiler(k, opts=f['opts'], includes=f['include'])
   group_2.append(t)
   all_targets.append(t)

# build it all
env.Alias(“all”, all_targets)

#build smaller sets
env.Alias(“group1”, group_1)
env.Alias(“group2”, group_2)


If you use Parts you would want to define the groups in to Parts files. I would suggest this simple sample: https://bitbucket.org/sconsparts/parts/src/6a57ea6ae48a/samples/subpart2/?at=master
As it shows making a tree with some independent targets that don’t directly depend on each other. Parts will load the all the file the first time, but after that it will avoid loading all the files given that target is not “all” or “.”

Hopefully this helps you with options to solve your issue. I think it is fair for Scons to take a few seconds to load everything when you have a few thousand files to as part of a builder. If I understand you setup correctly you are talking 2000 targets. In my case I had talked about before I had 1000 components, with each having a few to a couple thousand targets for each component ( most only had 15-20, there we had a few “mega” components defined 1000-+ files for a given builder. These just took time for SCons to process. I do think Parts might help if you have a good logic way to break up items into components. At the very least it will deal with the “target” management for you. However if most of you logic is like:

# compile a bunch of objects files with special flags
for k, f in group1:
   t = env.Object(k, opts=f['opts'], includes=f['include'])
   objs.append(t)
# link the as a program
env.Program(“myprogram”,objs)

I would suggest using the method Bill suggest with in a Part file, or via  pure Scons

Jason




From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Bill Deegan
Sent: Wednesday, August 16, 2017 9:49 AM
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] Help my build get faster... (was SCons and parts)

This should work.

for k, f in stuff_to_build:
   t = env.MyCompiler(k, opts=f['opts'], includes=f['include'])
   all_targets.append(t)

On Wed, Aug 16, 2017 at 2:48 AM, Pico Geyer <picogeyer at gmail.com<mailto:picogeyer at gmail.com>> wrote:
On Wed, Aug 16, 2017 at 4:56 AM, Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>> wrote:
> Pico,
>
> You can do this.
>
> env.MyBuilder(Target,Source, ANY_ENV_FLAG=['a','b','c'],
> CPPPATH=['z','d','f']
> There's no need to clone the environment to have specialized flags for a
> given builder or target..

Bill,

I think perhaps I didn't explain very well.
I don't need many flags for a single builder. I need a *different set*
of flags for each object.
so perhaps as an example, something like this:
stuff_to_build = {
'file1' : {opts = "-g -f", includes=['/path/to/include1', 'path/to/include2']},
'file2': { opts= "-g -h", includes=['/path/to/include3', 'path/to/include4']},
...
'file2000': {...}
}

And then the scons specific code would look like
for k, f in stuff_to_build:
   new_env = env.Clone()
   new_env['opts'] = f['opts']
   new_env['includes'] = f['includes']
   t = new_env.MyCompiler(k)
   all_targets.append(t)


So each object would be compiled as
mycompiler -o file1. out -g -f -I/path/to/include1 -I/path/to/include2 file1
mycompiler -o file2.out -g -h -I/path/to/include3 -I/path/to/include4 file2
...

So I really do need a different env for each object, right?
I know it may be a bit weird to have a different set of flags for each
object file but this is the requirement for that project and not
something that will change.

It's acceptable that scons take a few seconds on this mammoth project
but what really hurts is that when building something completely
unrelated (and that has no dependency on the mammoth project),  we
still have such a large startup time.
Now one solution would be completely disconnect the mammoth project
from the rest of the build. (I.E mv SConscript SConstruct)
Then I'd need to tell users that if they want to build something that
depends on the mammoth project then you need to first go to that
project and run scons there
and then come back and run the main scons build.
I'm still trying to avoid a solution like that because it adds more
required steps to our build, instead of just a simple scons command.

Sorry for the long story but I hope that clears things up?
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170816/fd3767be/attachment-0001.html>