[Scons-users] Hi, wondering about possible speed ups to scons

Thu Mar 12 17:11:38 EDT 2015

I believe the point of using watch dog like API is that it uses a system API to get what files changed. This is an event based API not a poll based. This would allow something like the below to get this effect without being editor dependent. I think the main logic loop in SCons would still need to see what need to rebuild. One file change could require a lot of work to be done. What is nice about this, is that when making such a change SCons could be rebuilding in the background while the user switched around or maybe makes a different change at a different level.

I will probably prototype this at some point in Parts once I get the code into bitbucket from tigris ( which I just got approval for ☺ ) and finish the current refactoring work on the Parts load logic.

Jason

From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Bill Deegan
Sent: Thursday, March 12, 2015 4:02 PM
To: SCons users mailing list
Subject: Re: [Scons-users] Hi, wondering about possible speed ups to scons

Steven Knight had done some work/thought about being able to directly tell SCons what files had changed, and having SCons "trust" that info and build according to that.
Such a functionality would be (IHMO) superior to just having SCons continuously look for changed files.  You could have your editor let an scons --interactive (perhaps captive) what files where just written..
-Bill

On Thu, Mar 12, 2015 at 1:59 PM, Kenny, Jason L <jason.l.kenny at intel.com<mailto:jason.l.kenny at intel.com>> wrote:
The interactive mode allows a user to load up a scontruct and have it build or clean without having to reload all the files. It used by many here at work for some of products we have need to load 1000+ build files. ( in this case they are Parts file.. ie my extension to SCons) Even with what I have done so far with Parts, it takes time ( ie 10 of seconds vs 100s of second) to load this information before Scons can start. The easy way to work around this is to use the –interactive mode in SCons. This will load everything up, then you say “build” or more correctly you give it a target and off SCons goes. This works well as once you make a change and you tell the interactive console to build it is off building in a second. Yes the interactive mode does not monitor for files changes or does a poll to see about rebuilding anything. This could be added at some point. The idea you suggest I think is called adding “build stream” support to SCons and I think it is a great idea to have added. However until that can happen the interactive mode should give you very fast turnaround time on rebuilds. Yes not as fast as having SCons responding to file change notification. But it should be good enough at the moment.

Jason

From: Scons-users [mailto:scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>] On Behalf Of Schleimer, Ben via Scons-users
Sent: Thursday, March 12, 2015 3:39 PM

To: SCons users mailing list
Subject: Re: [Scons-users] Hi, wondering about possible speed ups to scons

Hi Kenny,
   I don't understand how the --interactive mode is supposed to help.
It doesn't look like it monitors file changes...

To answer Erling's point, i wonder if the python module watchdog existed back when this concept was last explored? It seems to wrap a lot of the platform specific issues in a uniform manner.

Of course I am not a Scons expert (IANASE) so I don't know what the devils in the details are.

Cheers
Ben

On Thursday, March 12, 2015 8:41 AM, "Kenny, Jason L" <jason.l.kenny at intel.com<mailto:jason.l.kenny at intel.com>> wrote:

I don’t this idea will die. I think it just needs time to get done. I would start off with –interactive mode as this get most of the logic here needed for this as a user.

At the moment I think SCons is working more on certain core feature improvements just as python 3 support, toolchain and memory improvements. I think this idea is a good one.

Jason

From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Erling D. Andersen
Sent: Thursday, March 12, 2015 1:40 AM
To: SCons users mailing list
Subject: Re: [Scons-users] Hi, wondering about possible speed ups to scons

Hi

I have been Scons users for many years. That idea about using daemon has surfaced maybe 7 or 8 years. I am not sure why it died.
As far as I recall getting it to work cross platform was one issue. Without being an Scons expert I very much doubt you can get it working
without limiting what Scons currently can do.

On Wed, Mar 11, 2015 at 9:51 PM, Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>> wrote:
Ben,
While this sounds interesting (and useful in a code/compile/run/debug loop, but not really useful for other use models).
I believe it is way beyond the intended scope of the SCons project.
Have you tried scons --interactive ?
It's not quite what you are talking about, but should yield some of the benefits.

-Bill

On Wed, Mar 11, 2015 at 1:38 PM, Schleimer, Ben via Scons-users <scons-users at scons.org<mailto:scons-users at scons.org>> wrote:
Hi William,
   I was thinking that sconsd would keep an up-to-date scons diagraph in memory so that no time is spent re-calculating it during a compile.
Then when scons is run, it checks first for sconsd and only recalculates the digraph if sconsd is not running.

A further extension would be to do background compilation (like visual studio's C# system and eclipse's java)
That way, the user doesn't have to wait around for a large multi-file change compile...

Cheers
Ben

On Thursday, March 5, 2015 12:12 PM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:

Ben,

I'm interested to see your findings though I'm still not sure if I understand how the proposed sconsd is intended to behave.  Is the point to pre-compute md5sums or perhaps to keep an up-to-date scons digraph in memory?

V/R,
William

On Thu, Mar 5, 2015 at 2:45 PM, Schleimer, Ben via Scons-users <scons-users at scons.org<mailto:scons-users at scons.org>> wrote:
Hi Bill,
   It seems that inotify is supported in a limited manner on NFS on machines that support inotify (wow, that sounds like a self-proving statement)
http://unix.stackexchange.com/questions/81663/why-are-inotify-events-different-on-an-nfs-mount

However you are right, it would need to be tested. I'll try to do something like this and with huge numbers of files soonish just to see if its practical.

Cheers
Ben

On Thursday, March 5, 2015 7:49 AM, Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>> wrote:

To add to what Dirk said, you also need to test such on a non-local filesystem (NFS...)
-Bil

On Thu, Mar 5, 2015 at 3:27 AM, Dirk Bächle <tshortik at gmx.de<mailto:tshortik at gmx.de>> wrote:
Hi Ben,

On 05.03.2015 00:52, Schleimer, Ben via Scons-users wrote:
Hi Y'all,
    First of all, I'd like to say that I use SCons all of the time for my C++ projects and it's by far the best build system I've
had the pleasure to work with. I especially like the ease of using the Qt4 tools and precompiled header support, yada yada yada...

that's really nice to hear. Thanks a lot for the positive feedback!
Secondly, I've been thinking about all of the criticism aimed at scons for not being fast enough.
I understand that python is inheritantly slow (ignoring cython, etc..) but it seems like SCons could easily design it's way out of
this problem.

Did you have a look at http://www.scons.org/wiki/WhySconsIsNotSlow yet? Especially the numbers at the bottom of the page will be what you have to beat with your solution. There also exists a pre-edit version of the talk I gave at the PyconFr 2014 in Lyon (http://www.pycon.fr/2014/schedule/presentation/25/), the download URL is:

  http://dl.afpy.org/pycon-fr-14/videos/25-10-2014-112947_bd16.mp4
I'm wondering if the design change for SCons I'm about to propose has been proposed before:

Yes, it has...lots of times. ;) These kind of approaches always seem to work well, assuming that SCons has a DAG that is statically created in the background, and then simply traversed during the build step. But that's not true, SCons detects the dependencies on-the-fly while building targets. There is no explicit structure that you can point to and refer to as "the DAG". It's somehow implicitly given since each Node stores its children, and this is the info that gets updated continuously during the single build steps.
So, having a data structure as "the DAG" and update its nodes, will not directly map to what SCons uses internally to figure out which targets are up-to-date or need to get rebuilt.
1) Create a scons daemon (sconsd) that stays resident and watches a particular SConstruct file and all of its generated
dependencies. Whenever any of the dependencies change or is deleted, scons updates the dependency graph in the background.

2) When the user actually runs scons to compile, the dependency graph is ready to be immediately traversed and all of the Nodes built.

Thoughts? Ideas?

What about the Nodes that get generated during the build itself? What if I want to compile/build only a part of the complete DAG (a single subdirectory) and then later on possibly build the whole thing? How to ensure that things like build infos for the single Nodes/Targets get properly merged/updated between those two separate runs?

And how will such a daemon behave performance-wise on a project with 50000+ C files? My guess is that there would be a lot of stat()ing going on...and wrapped in a package like watchdog, one would have to make a simple testcase first and try to profile some decent numbers out of it. Then watch the scaling behaviour...and if that looks okay, we can talk about how to integrate such a thing into SCons. ;)

Just a few first thoughts...

Best regards,

Dirk

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

--
Erling D. Andersen

MOSEK APS
Fruebjergvej 3, Box 16,
DK-2100 Copenhagen O

Ph: +4539179907<tel:%2B4539179907>
Mo: +4523629520<tel:%2B4523629520>

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150312/502e9255/attachment-0001.html>