[Scons-users] Hi, wondering about possible speed ups to scons

[Dirk Bächle]

Hi Ben,

On 05.03.2015 00:52, Schleimer, Ben via Scons-users wrote:
> Hi Y'all,
>     First of all, I'd like to say that I use SCons all of the time for my C++ projects and it's by far the best build system I've
> had the pleasure to work with. I especially like the ease of using the Qt4 tools and precompiled header support, yada yada yada...
>

that's really nice to hear. Thanks a lot for the positive feedback!

> Secondly, I've been thinking about all of the criticism aimed at scons for not being fast enough.
> I understand that python is inheritantly slow (ignoring cython, etc..) but it seems like SCons could easily design it's way out of
> this problem.
>

Did you have a look at http://www.scons.org/wiki/WhySconsIsNotSlow yet? Especially the numbers at the bottom of the page will be 
what you have to beat with your solution. There also exists a pre-edit version of the talk I gave at the PyconFr 2014 in Lyon 
(http://www.pycon.fr/2014/schedule/presentation/25/), the download URL is:

   http://dl.afpy.org/pycon-fr-14/videos/25-10-2014-112947_bd16.mp4


> I'm wondering if the design change for SCons I'm about to propose has been proposed before:
>

Yes, it has...lots of times. ;) These kind of approaches always seem to work well, assuming that SCons has a DAG that is statically 
created in the background, and then simply traversed during the build step. But that's not true, SCons detects the dependencies 
on-the-fly while building targets. There is no explicit structure that you can point to and refer to as "the DAG". It's somehow 
implicitly given since each Node stores its children, and this is the info that gets updated continuously during the single build steps.
So, having a data structure as "the DAG" and update its nodes, will not directly map to what SCons uses internally to figure out 
which targets are up-to-date or need to get rebuilt.

> 1) Create a scons daemon (sconsd) that stays resident and watches a particular SConstruct file and all of its generated
> dependencies. Whenever any of the dependencies change or is deleted, scons updates the dependency graph in the background.
>
> 2) When the user actually runs scons to compile, the dependency graph is ready to be immediately traversed and all of the Nodes built.
>
> Thoughts? Ideas?
>

What about the Nodes that get generated during the build itself? What if I want to compile/build only a part of the complete DAG (a 
single subdirectory) and then later on possibly build the whole thing? How to ensure that things like build infos for the single 
Nodes/Targets get properly merged/updated between those two separate runs?

And how will such a daemon behave performance-wise on a project with 50000+ C files? My guess is that there would be a lot of 
stat()ing going on...and wrapped in a package like watchdog, one would have to make a simple testcase first and try to profile some 
decent numbers out of it. Then watch the scaling behaviour...and if that looks okay, we can talk about how to integrate such a thing 
into SCons. ;)

Just a few first thoughts...

Best regards,

Dirk