[Scons-users] Slowness of SCons with a large number of targets

[Jason Kenny]

The patch that should go into SCons came out of Part work. I believe the holdup was a desire for a fix to allow the use of a switch or something to control if is the old or new spawn logic was used. I personally think that is not useful as the current logic is going to be slow, and that everyone would want to have a posix_spawn logic for starting up a subprocess. Being forced to fork GB or memory does not make any sense in the case of the build system. This change made a huge difference on Linux based builds. For some reason newer Linux kernel seem to benefit from this more than the older ones.

In theory just including Parts in  the Sconstruct should do all the magic for you. However I have not tested this well enough, so there might be a different issue that would show up.

Jason

-----Original Message-----
From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Hans Ottevanger
Sent: Tuesday, April 18, 2017 6:48 AM
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] Slowness of SCons with a large number of targets

Thanks for the response.

We will first see if using the stubwrapper, as suggested by Bill Keegan, will bring any relief. Is that in any way related to the subprocess fix in Parts?

Since incremental builds are also quite crucial for our organisation, we could try Parts later, after we tried the stubwrapper.

Best regards,

Hans Ottevanger

On Sat, Apr 15, 2017 at 2:41 AM, Jason Kenny <dragon512 at live.com> wrote:
> You can pull in Parts. It should help in two different ways.
>
> 1) it has a fix to subprocess that should help address the main issues you have.
> 2) is you use the "part()" concept as well, Parts will help on incremental builds buy reducing what is loaded in memory. This will greatly speed up the builds and time to first build actions.
>
> Hope this helps
>
> Jason
>
> -----Original Message-----
> From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of 
> Hans Ottevanger
> Sent: Friday, April 14, 2017 5:54 AM
> To: scons-users at scons.org
> Subject: [Scons-users] Slowness of SCons with a large number of 
> targets
>
> Hi,
>
> At the company I am working for we are revising our software build system and we are currently evaluating build tools. We think that SCons is an interesting tool, offering a high degree of consistency and features that others are clearly missing.
>
> However, we are facing severe performance issues with an increasing number of targets. Experimenting with close to 300000 targets in a tree that mimics a large part of our actual tree, we measured times for a full build of about 35 hours. We just touched the target files and did not invoke real compilers, so those 35 hours are mainly overhead from using SCons.
>
> We are aware of the fact that Eric Melski already reported scalability issues quite some time ago (see https://blog.melski.net/2013/12/11/update-scons-is-still-really-slow/).
> We could almost exactly reproduce Erics results using the tools he 
> provides on GitHub (https://github.com/emelski/scons_bench). We are 
> using SCons 2.5.1 and Python 2.7 on VDIs with 4 cores and 16GB RAM and
> (perceived) local disk storage, running RHEL6. We need 5000 seconds for 50000 targets, but we see the same quadratic behaviour as Eric Melski reports.
>
> We understand that the issue was diagnosed as being caused by the way Python implements fork() and waitpid() and that relief was expected from a wrapper using posix_spawn(). That stubprocess.py wrapper was slated for inclusion in SCons 2.5, but apparently did not make it (see https://bitbucket.org/scons/scons/wiki/Roadmap).
>
> What are the current plans integrating this stubprocess.py wrapper into an SCons release? And is there already an estimate when we can expect that?
>
> Best regards,
>
> Hans Ottevanger
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users