[Scons-users] Need help to improve the incremental build time while using SCons

[Bill Deegan]

Things which can speed up null builds is to make sure that:
a) you're not creating tons of Environment()'s  - That's a common rookie
mistake
b) you're only loading the tools you need via Environment()'s tool=[] list
(the default list is likely more than you may need)
c) as Dirk said --interactive can help
d) --implicit-cache can help.
e) are you using Configure contexts? If so, some way to avoid rerunning
that logic when rerunning in the same environment might help.

I have some more ideas to speed up some critical path logic, but time will
tell if that pans out.  Profiling looks like some changes could help with
Subst() calls.

Also, in the case of a null build, we still write out the sconsign, on the
fairly large build I'm working on, that takes about 10 out of 55 seconds
for a null build.
So figuring out a way to avoid writing when nothing is built will help, but
only if nothing is out of date.
I've seen some information suggesting that JSON write/read is much faster
than pickle (which is what's in the sconsign files).  That'll take some
effort to try out.

There may be some more heavyweight changes which can help, but those'll
wait until I exhaust the easier things to try.

-Bill
SCons Project Co-Manager



On Tue, Jun 20, 2017 at 2:49 PM, Dirk Bächle <tshortik at gmx.de> wrote:

> Hi Basil,
>
>
> I'd like to second Bill here
>
> On 20.06.2017 17:02, Bill Deegan wrote:
>
>
> [...]
>
>
>
> To some extent SCons will always be slower than Make as it is doing more
> to ensure that files are rebuilt when needed.
> (Make doesn't check if command lines for compilation changes, or if the
> compiler changes, and many other things which should cause a rebuild).
>
>
> . I've done quite some profiling of the SCons core in the past (see
> https://bitbucket.org/scons/scons/wiki/WhySconsIsNotSlow ), and it's not
> possible to find a single spot for an "update run" where all the runtime is
> spent. Then it would be easy to make things faster with little effort.
> Instead the runtime is spread over numerous methods and routines. On
> closer inspection, all this "overhead" is needed because it offers the kind
> of flexibility and extensibility that SCons is really about.
> One could shortcut certain operations, but not without "crippling" the
> core's functionality in more or less drastic ways.
>
> Having said this, as a first step you may want to call SCons with the
> "--debug=time" option. This will give you an impression about how much time
> is consumed for reading the SConscripts (not a lot of speedup possible
> here, unless something is really, really wrong with how you setup your
> build) vs the actual "check-whether-everything-is-up-to-date" time.
> A large part of the latter will be spent on substituting string values and
> computing build signatures. You could try to make things faster by using
> the "fastcpp" Tool that I wrote (see https://bitbucket.org/scons/
> scons/wiki/ToolsIndex or https://bitbucket.org/dirkbaechle/scons_fastcpp
> directly). It will pre-expand certain string variables and this can save up
> to 20% update time. It also disables a default feature of SCons that tries
> to find the used compiler/tools executables in the $PATH and caches their
> MD5 hash as well.
> Depending on your $PATH setting, this can already help a lot.
>
> Using a different Decider, as being proposed very often, won't help at
> all. SCons internally *always* computes and updates the MD5 hashes
> too...such that no complete rebuild is required when switching from one
> Decider type to another on two consecutive builds.
>
> For even faster edit-compile cycles you should try to avoid a simple
> "scons", which tries to build everything. If you simply know that your
> latest changes have been to the "foomagic" program (or a library that's
> required), a "scons foomagic" will give you a drastic speedup. This is
> because now SCons will only expand and trace back the dependencies for the
> target (folder) you specified.
> This works even if the required dependencies for "foomagic" aren't in the
> same folder (or below it in the directory hierarchy).
>
> Finally, try the "interactive" mode, which let's you avoid the cost of
> re-reading all the SConscripts on every build/update. Combined with
> specifying a concrete target (folder), I was able to run a "no changes
> update" for a library of 1000 files in a 100k file project in about 1-2
> seconds.
>
> I hope this all gives you some further ideas.
>
> Best regards,
>
> Dirk
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170620/9a9f13b2/attachment-0001.html>