[Scons-users] Hi, wondering about possible speed ups to scons

Schleimer, Ben bensch128 at yahoo.com
Wed Mar 11 16:38:53 EDT 2015

Hi William, 
   I was thinking that sconsd would keep an up-to-date scons diagraph in memory so that no time is spent re-calculating it during a compile.Then when scons is run, it checks first for sconsd and only recalculates the digraph if sconsd is not running.

A further extension would be to do background compilation (like visual studio's C# system and eclipse's java) 
That way, the user doesn't have to wait around for a large multi-file change compile...

     On Thursday, March 5, 2015 12:12 PM, William Blevins <wblevins001 at gmail.com> wrote:

I'm interested to see your findings though I'm still not sure if I understand how the proposed sconsd is intended to behave.  Is the point to pre-compute md5sums or perhaps to keep an up-to-date scons digraph in memory?
On Thu, Mar 5, 2015 at 2:45 PM, Schleimer, Ben via Scons-users <scons-users at scons.org> wrote:

Hi Bill,
   It seems that inotify is supported in a limited manner on NFS on machines that support inotify (wow, that sounds like a self-proving statement)
However you are right, it would need to be tested. I'll try to do something like this and with huge numbers of files soonish just to see if its practical.

     On Thursday, March 5, 2015 7:49 AM, Bill Deegan <bill at baddogconsulting.com> wrote:

 To add to what Dirk said, you also need to test such on a non-local filesystem (NFS...)

On Thu, Mar 5, 2015 at 3:27 AM, Dirk Bächle <tshortik at gmx.de> wrote:

Hi Ben,

On 05.03.2015 00:52, Schleimer, Ben via Scons-users wrote:

Hi Y'all,
    First of all, I'd like to say that I use SCons all of the time for my C++ projects and it's by far the best build system I've
had the pleasure to work with. I especially like the ease of using the Qt4 tools and precompiled header support, yada yada yada...

that's really nice to hear. Thanks a lot for the positive feedback!

Secondly, I've been thinking about all of the criticism aimed at scons for not being fast enough.
I understand that python is inheritantly slow (ignoring cython, etc..) but it seems like SCons could easily design it's way out of
this problem.

Did you have a look at http://www.scons.org/wiki/WhySconsIsNotSlow yet? Especially the numbers at the bottom of the page will be what you have to beat with your solution. There also exists a pre-edit version of the talk I gave at the PyconFr 2014 in Lyon (http://www.pycon.fr/2014/schedule/presentation/25/), the download URL is:


I'm wondering if the design change for SCons I'm about to propose has been proposed before:

Yes, it has...lots of times. ;) These kind of approaches always seem to work well, assuming that SCons has a DAG that is statically created in the background, and then simply traversed during the build step. But that's not true, SCons detects the dependencies on-the-fly while building targets. There is no explicit structure that you can point to and refer to as "the DAG". It's somehow implicitly given since each Node stores its children, and this is the info that gets updated continuously during the single build steps.
So, having a data structure as "the DAG" and update its nodes, will not directly map to what SCons uses internally to figure out which targets are up-to-date or need to get rebuilt.

1) Create a scons daemon (sconsd) that stays resident and watches a particular SConstruct file and all of its generated
dependencies. Whenever any of the dependencies change or is deleted, scons updates the dependency graph in the background.

2) When the user actually runs scons to compile, the dependency graph is ready to be immediately traversed and all of the Nodes built.

Thoughts? Ideas?

What about the Nodes that get generated during the build itself? What if I want to compile/build only a part of the complete DAG (a single subdirectory) and then later on possibly build the whole thing? How to ensure that things like build infos for the single Nodes/Targets get properly merged/updated between those two separate runs?

And how will such a daemon behave performance-wise on a project with 50000+ C files? My guess is that there would be a lot of stat()ing going on...and wrapped in a package like watchdog, one would have to make a simple testcase first and try to profile some decent numbers out of it. Then watch the scaling behaviour...and if that looks okay, we can talk about how to integrate such a thing into SCons. ;)

Just a few first thoughts...

Best regards,


Scons-users mailing list
Scons-users at scons.org

Scons-users mailing list
Scons-users at scons.org

Scons-users mailing list
Scons-users at scons.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150311/2b78b744/attachment.html>

More information about the Scons-users mailing list