[Scons-users] Hi, wondering about possible speed ups to scons

Thu Mar 12 02:39:50 EDT 2015

Hi

I have been Scons users for many years. That idea about using daemon has
surfaced maybe 7 or 8 years. I am not sure why it died.
As far as I recall getting it to work cross platform was one issue. Without
being an Scons expert I very much doubt you can get it working
without limiting what Scons currently can do.

On Wed, Mar 11, 2015 at 9:51 PM, Bill Deegan <bill at baddogconsulting.com>
wrote:

> Ben,
>
> While this sounds interesting (and useful in a code/compile/run/debug
> loop, but not really useful for other use models).
> I believe it is way beyond the intended scope of the SCons project.
>
> Have you tried scons --interactive ?
> It's not quite what you are talking about, but should yield some of the
> benefits.
>
> -Bill
>
> On Wed, Mar 11, 2015 at 1:38 PM, Schleimer, Ben via Scons-users <
> scons-users at scons.org> wrote:
>
>> Hi William,
>>    I was thinking that sconsd would keep an up-to-date scons diagraph in
>> memory so that no time is spent re-calculating it during a compile.
>> Then when scons is run, it checks first for sconsd and only recalculates
>> the digraph if sconsd is not running.
>>
>> A further extension would be to do background compilation (like visual
>> studio's C# system and eclipse's java)
>> That way, the user doesn't have to wait around for a large multi-file
>> change compile...
>>
>> Cheers
>> Ben
>>
>>
>>   On Thursday, March 5, 2015 12:12 PM, William Blevins <
>> wblevins001 at gmail.com> wrote:
>>
>>
>>
>> Ben,
>>
>> I'm interested to see your findings though I'm still not sure if I
>> understand how the proposed sconsd is intended to behave.  Is the point to
>> pre-compute md5sums or perhaps to keep an up-to-date scons digraph in
>> memory?
>>
>> V/R,
>> William
>>
>> On Thu, Mar 5, 2015 at 2:45 PM, Schleimer, Ben via Scons-users <
>> scons-users at scons.org> wrote:
>>
>> Hi Bill,
>>    It seems that inotify is supported in a limited manner on NFS on
>> machines that support inotify (wow, that sounds like a self-proving
>> statement)
>>
>> http://unix.stackexchange.com/questions/81663/why-are-inotify-events-different-on-an-nfs-mount
>>
>> However you are right, it would need to be tested. I'll try to do
>> something like this and with huge numbers of files soonish just to see if
>> its practical.
>>
>> Cheers
>> Ben
>>
>>   On Thursday, March 5, 2015 7:49 AM, Bill Deegan <
>> bill at baddogconsulting.com> wrote:
>>
>>
>>
>> To add to what Dirk said, you also need to test such on a non-local
>> filesystem (NFS...)
>>
>> -Bil
>>
>> On Thu, Mar 5, 2015 at 3:27 AM, Dirk Bächle <tshortik at gmx.de> wrote:
>>
>> Hi Ben,
>>
>> On 05.03.2015 00:52, Schleimer, Ben via Scons-users wrote:
>>
>> Hi Y'all,
>>     First of all, I'd like to say that I use SCons all of the time for my
>> C++ projects and it's by far the best build system I've
>> had the pleasure to work with. I especially like the ease of using the
>> Qt4 tools and precompiled header support, yada yada yada...
>>
>>
>> that's really nice to hear. Thanks a lot for the positive feedback!
>>
>>  Secondly, I've been thinking about all of the criticism aimed at scons
>> for not being fast enough.
>> I understand that python is inheritantly slow (ignoring cython, etc..)
>> but it seems like SCons could easily design it's way out of
>> this problem.
>>
>>
>> Did you have a look at http://www.scons.org/wiki/WhySconsIsNotSlow yet?
>> Especially the numbers at the bottom of the page will be what you have to
>> beat with your solution. There also exists a pre-edit version of the talk I
>> gave at the PyconFr 2014 in Lyon (http://www.pycon.fr/2014/
>> schedule/presentation/25/), the download URL is:
>>
>>   http://dl.afpy.org/pycon-fr-14/videos/25-10-2014-112947_bd16.mp4
>>
>>
>>  I'm wondering if the design change for SCons I'm about to propose has
>> been proposed before:
>>
>>
>> Yes, it has...lots of times. ;) These kind of approaches always seem to
>> work well, assuming that SCons has a DAG that is statically created in the
>> background, and then simply traversed during the build step. But that's not
>> true, SCons detects the dependencies on-the-fly while building targets.
>> There is no explicit structure that you can point to and refer to as "the
>> DAG". It's somehow implicitly given since each Node stores its children,
>> and this is the info that gets updated continuously during the single build
>> steps.
>> So, having a data structure as "the DAG" and update its nodes, will not
>> directly map to what SCons uses internally to figure out which targets are
>> up-to-date or need to get rebuilt.
>>
>>  1) Create a scons daemon (sconsd) that stays resident and watches a
>> particular SConstruct file and all of its generated
>> dependencies. Whenever any of the dependencies change or is deleted,
>> scons updates the dependency graph in the background.
>>
>> 2) When the user actually runs scons to compile, the dependency graph is
>> ready to be immediately traversed and all of the Nodes built.
>>
>> Thoughts? Ideas?
>>
>>
>> What about the Nodes that get generated during the build itself? What if
>> I want to compile/build only a part of the complete DAG (a single
>> subdirectory) and then later on possibly build the whole thing? How to
>> ensure that things like build infos for the single Nodes/Targets get
>> properly merged/updated between those two separate runs?
>>
>> And how will such a daemon behave performance-wise on a project with
>> 50000+ C files? My guess is that there would be a lot of stat()ing going
>> on...and wrapped in a package like watchdog, one would have to make a
>> simple testcase first and try to profile some decent numbers out of it.
>> Then watch the scaling behaviour...and if that looks okay, we can talk
>> about how to integrate such a thing into SCons. ;)
>>
>> Just a few first thoughts...
>>
>> Best regards,
>>
>> Dirk
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>

-- 
Erling D. Andersen

MOSEK APS
Fruebjergvej 3, Box 16,
DK-2100 Copenhagen O

Ph: +4539179907
Mo: +4523629520
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150312/be6dce4c/attachment.html>