[Scons-users] Hi, wondering about possible speed ups to scons

Thu Mar 12 11:41:24 EDT 2015

I don’t this idea will die. I think it just needs time to get done. I would start off with –interactive mode as this get most of the logic here needed for this as a user.

At the moment I think SCons is working more on certain core feature improvements just as python 3 support, toolchain and memory improvements. I think this idea is a good one.

Jason

From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Erling D. Andersen
Sent: Thursday, March 12, 2015 1:40 AM
To: SCons users mailing list
Subject: Re: [Scons-users] Hi, wondering about possible speed ups to scons

Hi

I have been Scons users for many years. That idea about using daemon has surfaced maybe 7 or 8 years. I am not sure why it died.
As far as I recall getting it to work cross platform was one issue. Without being an Scons expert I very much doubt you can get it working
without limiting what Scons currently can do.

On Wed, Mar 11, 2015 at 9:51 PM, Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>> wrote:
Ben,
While this sounds interesting (and useful in a code/compile/run/debug loop, but not really useful for other use models).
I believe it is way beyond the intended scope of the SCons project.
Have you tried scons --interactive ?
It's not quite what you are talking about, but should yield some of the benefits.

-Bill

On Wed, Mar 11, 2015 at 1:38 PM, Schleimer, Ben via Scons-users <scons-users at scons.org<mailto:scons-users at scons.org>> wrote:
Hi William,
   I was thinking that sconsd would keep an up-to-date scons diagraph in memory so that no time is spent re-calculating it during a compile.
Then when scons is run, it checks first for sconsd and only recalculates the digraph if sconsd is not running.

A further extension would be to do background compilation (like visual studio's C# system and eclipse's java)
That way, the user doesn't have to wait around for a large multi-file change compile...

Cheers
Ben

On Thursday, March 5, 2015 12:12 PM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:

Ben,

I'm interested to see your findings though I'm still not sure if I understand how the proposed sconsd is intended to behave.  Is the point to pre-compute md5sums or perhaps to keep an up-to-date scons digraph in memory?

V/R,
William

On Thu, Mar 5, 2015 at 2:45 PM, Schleimer, Ben via Scons-users <scons-users at scons.org<mailto:scons-users at scons.org>> wrote:
Hi Bill,
   It seems that inotify is supported in a limited manner on NFS on machines that support inotify (wow, that sounds like a self-proving statement)
http://unix.stackexchange.com/questions/81663/why-are-inotify-events-different-on-an-nfs-mount

However you are right, it would need to be tested. I'll try to do something like this and with huge numbers of files soonish just to see if its practical.

Cheers
Ben

On Thursday, March 5, 2015 7:49 AM, Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>> wrote:

To add to what Dirk said, you also need to test such on a non-local filesystem (NFS...)
-Bil

On Thu, Mar 5, 2015 at 3:27 AM, Dirk Bächle <tshortik at gmx.de<mailto:tshortik at gmx.de>> wrote:
Hi Ben,

On 05.03.2015 00:52, Schleimer, Ben via Scons-users wrote:
Hi Y'all,
    First of all, I'd like to say that I use SCons all of the time for my C++ projects and it's by far the best build system I've
had the pleasure to work with. I especially like the ease of using the Qt4 tools and precompiled header support, yada yada yada...

that's really nice to hear. Thanks a lot for the positive feedback!
Secondly, I've been thinking about all of the criticism aimed at scons for not being fast enough.
I understand that python is inheritantly slow (ignoring cython, etc..) but it seems like SCons could easily design it's way out of
this problem.

Did you have a look at http://www.scons.org/wiki/WhySconsIsNotSlow yet? Especially the numbers at the bottom of the page will be what you have to beat with your solution. There also exists a pre-edit version of the talk I gave at the PyconFr 2014 in Lyon (http://www.pycon.fr/2014/schedule/presentation/25/), the download URL is:

  http://dl.afpy.org/pycon-fr-14/videos/25-10-2014-112947_bd16.mp4

I'm wondering if the design change for SCons I'm about to propose has been proposed before:

Yes, it has...lots of times. ;) These kind of approaches always seem to work well, assuming that SCons has a DAG that is statically created in the background, and then simply traversed during the build step. But that's not true, SCons detects the dependencies on-the-fly while building targets. There is no explicit structure that you can point to and refer to as "the DAG". It's somehow implicitly given since each Node stores its children, and this is the info that gets updated continuously during the single build steps.
So, having a data structure as "the DAG" and update its nodes, will not directly map to what SCons uses internally to figure out which targets are up-to-date or need to get rebuilt.
1) Create a scons daemon (sconsd) that stays resident and watches a particular SConstruct file and all of its generated
dependencies. Whenever any of the dependencies change or is deleted, scons updates the dependency graph in the background.

2) When the user actually runs scons to compile, the dependency graph is ready to be immediately traversed and all of the Nodes built.

Thoughts? Ideas?

What about the Nodes that get generated during the build itself? What if I want to compile/build only a part of the complete DAG (a single subdirectory) and then later on possibly build the whole thing? How to ensure that things like build infos for the single Nodes/Targets get properly merged/updated between those two separate runs?

And how will such a daemon behave performance-wise on a project with 50000+ C files? My guess is that there would be a lot of stat()ing going on...and wrapped in a package like watchdog, one would have to make a simple testcase first and try to profile some decent numbers out of it. Then watch the scaling behaviour...and if that looks okay, we can talk about how to integrate such a thing into SCons. ;)

Just a few first thoughts...

Best regards,

Dirk

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

--
Erling D. Andersen

MOSEK APS
Fruebjergvej 3, Box 16,
DK-2100 Copenhagen O

Ph: +4539179907
Mo: +4523629520

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150312/af3b72aa/attachment-0001.html>