[Scons-users] Hi, wondering about possible speed ups to scons

Thu Mar 12 17:02:07 EDT 2015

Steven Knight had done some work/thought about being able to directly tell
SCons what files had changed, and having SCons "trust" that info and build
according to that.

Such a functionality would be (IHMO) superior to just having SCons
continuously look for changed files.  You could have your editor let an
scons --interactive (perhaps captive) what files where just written..

-Bill

On Thu, Mar 12, 2015 at 1:59 PM, Kenny, Jason L <jason.l.kenny at intel.com>
wrote:

>  The interactive mode allows a user to load up a scontruct and have it
> build or clean without having to reload all the files. It used by many here
> at work for some of products we have need to load 1000+ build files. ( in
> this case they are Parts file.. ie my extension to SCons) Even with what I
> have done so far with Parts, it takes time ( ie 10 of seconds vs 100s of
> second) to load this information before Scons can start. The easy way to
> work around this is to use the –interactive mode in SCons. This will load
> everything up, then you say “build” or more correctly you give it a target
> and off SCons goes. This works well as once you make a change and you tell
> the interactive console to build it is off building in a second. Yes the
> interactive mode does not monitor for files changes or does a poll to see
> about rebuilding anything. This could be added at some point. The idea you
> suggest I think is called adding “build stream” support to SCons and I
> think it is a great idea to have added. However until that can happen the
> interactive mode should give you very fast turnaround time on rebuilds. Yes
> not as fast as having SCons responding to file change notification. But it
> should be good enough at the moment.
>
>
>
> Jason
>
>
>
>
>
> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf Of *Schleimer,
> Ben via Scons-users
> *Sent:* Thursday, March 12, 2015 3:39 PM
>
> *To:* SCons users mailing list
> *Subject:* Re: [Scons-users] Hi, wondering about possible speed ups to
> scons
>
>
>
> Hi Kenny,
>
>    I don't understand how the --interactive mode is supposed to help.
>
> It doesn't look like it monitors file changes...
>
>
>
> To answer Erling's point, i wonder if the python module watchdog existed
> back when this concept was last explored? It seems to wrap a lot of the
> platform specific issues in a uniform manner.
>
>
>
> Of course I am not a Scons expert (IANASE) so I don't know what the devils
> in the details are.
>
>
>
> Cheers
>
> Ben
>
>
>
> On Thursday, March 12, 2015 8:41 AM, "Kenny, Jason L" <
> jason.l.kenny at intel.com> wrote:
>
>
>
> I don’t this idea will die. I think it just needs time to get done. I
> would start off with –interactive mode as this get most of the logic here
> needed for this as a user.
>
>
>
> At the moment I think SCons is working more on certain core feature
> improvements just as python 3 support, toolchain and memory improvements. I
> think this idea is a good one.
>
>
>
> Jason
>
>
>
> *From:* Scons-users [mailto:scons-users-bounces at scons.org
> <scons-users-bounces at scons.org>] *On Behalf Of *Erling D. Andersen
> *Sent:* Thursday, March 12, 2015 1:40 AM
> *To:* SCons users mailing list
> *Subject:* Re: [Scons-users] Hi, wondering about possible speed ups to
> scons
>
>
>
> Hi
>
>
>
> I have been Scons users for many years. That idea about using daemon has
> surfaced maybe 7 or 8 years. I am not sure why it died.
>
> As far as I recall getting it to work cross platform was one issue.
> Without being an Scons expert I very much doubt you can get it working
>
> without limiting what Scons currently can do.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Mar 11, 2015 at 9:51 PM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>    Ben,
>
> While this sounds interesting (and useful in a code/compile/run/debug
> loop, but not really useful for other use models).
>
> I believe it is way beyond the intended scope of the SCons project.
>
> Have you tried scons --interactive ?
>
> It's not quite what you are talking about, but should yield some of the
> benefits.
>
>
>
> -Bill
>
>
>
> On Wed, Mar 11, 2015 at 1:38 PM, Schleimer, Ben via Scons-users <
> scons-users at scons.org> wrote:
>
>   Hi William,
>
>    I was thinking that sconsd would keep an up-to-date scons diagraph in
> memory so that no time is spent re-calculating it during a compile.
>
> Then when scons is run, it checks first for sconsd and only recalculates
> the digraph if sconsd is not running.
>
>
>
> A further extension would be to do background compilation (like visual
> studio's C# system and eclipse's java)
>
> That way, the user doesn't have to wait around for a large multi-file
> change compile...
>
>
>
> Cheers
>
> Ben
>
>
>
> On Thursday, March 5, 2015 12:12 PM, William Blevins <
> wblevins001 at gmail.com> wrote:
>
>
>
> Ben,
>
>
>
> I'm interested to see your findings though I'm still not sure if I
> understand how the proposed sconsd is intended to behave.  Is the point to
> pre-compute md5sums or perhaps to keep an up-to-date scons digraph in
> memory?
>
>
>
> V/R,
>
> William
>
>
>
> On Thu, Mar 5, 2015 at 2:45 PM, Schleimer, Ben via Scons-users <
> scons-users at scons.org> wrote:
>
>    Hi Bill,
>
>    It seems that inotify is supported in a limited manner on NFS on
> machines that support inotify (wow, that sounds like a self-proving
> statement)
>
>
> http://unix.stackexchange.com/questions/81663/why-are-inotify-events-different-on-an-nfs-mount
>
>
>
> However you are right, it would need to be tested. I'll try to do
> something like this and with huge numbers of files soonish just to see if
> its practical.
>
>
>
> Cheers
>
> Ben
>
>
>
> On Thursday, March 5, 2015 7:49 AM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>
>
> To add to what Dirk said, you also need to test such on a non-local
> filesystem (NFS...)
>
> -Bil
>
>
>
> On Thu, Mar 5, 2015 at 3:27 AM, Dirk Bächle <tshortik at gmx.de> wrote:
>
>  Hi Ben,
>
> On 05.03.2015 00:52, Schleimer, Ben via Scons-users wrote:
>
>  Hi Y'all,
>     First of all, I'd like to say that I use SCons all of the time for my
> C++ projects and it's by far the best build system I've
> had the pleasure to work with. I especially like the ease of using the Qt4
> tools and precompiled header support, yada yada yada...
>
>
> that's really nice to hear. Thanks a lot for the positive feedback!
>
>  Secondly, I've been thinking about all of the criticism aimed at scons
> for not being fast enough.
> I understand that python is inheritantly slow (ignoring cython, etc..) but
> it seems like SCons could easily design it's way out of
> this problem.
>
>
> Did you have a look at http://www.scons.org/wiki/WhySconsIsNotSlow yet?
> Especially the numbers at the bottom of the page will be what you have to
> beat with your solution. There also exists a pre-edit version of the talk I
> gave at the PyconFr 2014 in Lyon (
> http://www.pycon.fr/2014/schedule/presentation/25/), the download URL is:
>
>   http://dl.afpy.org/pycon-fr-14/videos/25-10-2014-112947_bd16.mp4
>
>  I'm wondering if the design change for SCons I'm about to propose has
> been proposed before:
>
>
> Yes, it has...lots of times. ;) These kind of approaches always seem to
> work well, assuming that SCons has a DAG that is statically created in the
> background, and then simply traversed during the build step. But that's not
> true, SCons detects the dependencies on-the-fly while building targets.
> There is no explicit structure that you can point to and refer to as "the
> DAG". It's somehow implicitly given since each Node stores its children,
> and this is the info that gets updated continuously during the single build
> steps.
> So, having a data structure as "the DAG" and update its nodes, will not
> directly map to what SCons uses internally to figure out which targets are
> up-to-date or need to get rebuilt.
>
>  1) Create a scons daemon (sconsd) that stays resident and watches a
> particular SConstruct file and all of its generated
> dependencies. Whenever any of the dependencies change or is deleted, scons
> updates the dependency graph in the background.
>
> 2) When the user actually runs scons to compile, the dependency graph is
> ready to be immediately traversed and all of the Nodes built.
>
> Thoughts? Ideas?
>
>
> What about the Nodes that get generated during the build itself? What if I
> want to compile/build only a part of the complete DAG (a single
> subdirectory) and then later on possibly build the whole thing? How to
> ensure that things like build infos for the single Nodes/Targets get
> properly merged/updated between those two separate runs?
>
> And how will such a daemon behave performance-wise on a project with
> 50000+ C files? My guess is that there would be a lot of stat()ing going
> on...and wrapped in a package like watchdog, one would have to make a
> simple testcase first and try to profile some decent numbers out of it.
> Then watch the scaling behaviour...and if that looks okay, we can talk
> about how to integrate such a thing into SCons. ;)
>
> Just a few first thoughts...
>
> Best regards,
>
> Dirk
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
>
> --
>
> Erling D. Andersen
>
>
>
> MOSEK APS
>
> Fruebjergvej 3, Box 16,
>
> DK-2100 Copenhagen O
>
>
>
> Ph: +4539179907
>
> Mo: +4523629520
>
>
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150312/5e66fa66/attachment-0001.html>