[Scons-users] Hi, wondering about possible speed ups to scons

[Schleimer, Ben]

Hi Bill,
   It seems that inotify is supported in a limited manner on NFS on machines that support inotify (wow, that sounds like a self-proving statement)
http://unix.stackexchange.com/questions/81663/why-are-inotify-events-different-on-an-nfs-mount
However you are right, it would need to be tested. I'll try to do something like this and with huge numbers of files soonish just to see if its practical.
CheersBen

     On Thursday, March 5, 2015 7:49 AM, Bill Deegan <bill at baddogconsulting.com> wrote:
   
 

 To add to what Dirk said, you also need to test such on a non-local filesystem (NFS...)

-Bil
On Thu, Mar 5, 2015 at 3:27 AM, Dirk Bächle <tshortik at gmx.de> wrote:

Hi Ben,

On 05.03.2015 00:52, Schleimer, Ben via Scons-users wrote:

Hi Y'all,
    First of all, I'd like to say that I use SCons all of the time for my C++ projects and it's by far the best build system I've
had the pleasure to work with. I especially like the ease of using the Qt4 tools and precompiled header support, yada yada yada...



that's really nice to hear. Thanks a lot for the positive feedback!


Secondly, I've been thinking about all of the criticism aimed at scons for not being fast enough.
I understand that python is inheritantly slow (ignoring cython, etc..) but it seems like SCons could easily design it's way out of
this problem.



Did you have a look at http://www.scons.org/wiki/WhySconsIsNotSlow yet? Especially the numbers at the bottom of the page will be what you have to beat with your solution. There also exists a pre-edit version of the talk I gave at the PyconFr 2014 in Lyon (http://www.pycon.fr/2014/schedule/presentation/25/), the download URL is:

  http://dl.afpy.org/pycon-fr-14/videos/25-10-2014-112947_bd16.mp4



I'm wondering if the design change for SCons I'm about to propose has been proposed before:



Yes, it has...lots of times. ;) These kind of approaches always seem to work well, assuming that SCons has a DAG that is statically created in the background, and then simply traversed during the build step. But that's not true, SCons detects the dependencies on-the-fly while building targets. There is no explicit structure that you can point to and refer to as "the DAG". It's somehow implicitly given since each Node stores its children, and this is the info that gets updated continuously during the single build steps.
So, having a data structure as "the DAG" and update its nodes, will not directly map to what SCons uses internally to figure out which targets are up-to-date or need to get rebuilt.


1) Create a scons daemon (sconsd) that stays resident and watches a particular SConstruct file and all of its generated
dependencies. Whenever any of the dependencies change or is deleted, scons updates the dependency graph in the background.

2) When the user actually runs scons to compile, the dependency graph is ready to be immediately traversed and all of the Nodes built.

Thoughts? Ideas?



What about the Nodes that get generated during the build itself? What if I want to compile/build only a part of the complete DAG (a single subdirectory) and then later on possibly build the whole thing? How to ensure that things like build infos for the single Nodes/Targets get properly merged/updated between those two separate runs?

And how will such a daemon behave performance-wise on a project with 50000+ C files? My guess is that there would be a lot of stat()ing going on...and wrapped in a package like watchdog, one would have to make a simple testcase first and try to profile some decent numbers out of it. Then watch the scaling behaviour...and if that looks okay, we can talk about how to integrate such a thing into SCons. ;)

Just a few first thoughts...

Best regards,

Dirk

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users



_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users


 
   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150305/73f396d3/attachment.html>