[Scons-users] Announcement: SCons core changes ahead...

[Kenny, Jason L]

Hi,

Just to add a little here. I don't think there will be any change in the spawn logic. The monkey patch to subprocess we made in Parts and submitted to SCons is about changing the fork exec logic to vfork exec ( or posix_spawn). This prevents the system from having to set GBs of pages to copy on write. It was found that as the SCons build get larger the system get a heavy load of having to reset a large amount of memory that is never shared to copy on write. What is worse is that whenever SCons spawns an action it generally updates some internal state before the task master spawn a new command. This forces page copies to happen in the SCons process, slowing it down even more. For example we had a local build of a product that on RHEL 4 took 2 hours, when it move to RHEL 5 it took 4 hours. It was found via some custom profiling tools we made that the issue was with system spawning. Tweaks to spawning logic reduced the RHEL5 build to 2 hours and when tested on RHEL 4 it was down to about an hour. I still don't know what changed in the kernel to make RHEL5 so much slower vs rhel4 ....

At anyrate if you call subprocess with in SCons to spawn a command via overriding the SPAWN variable ( as I do in Parts) this improvement will be automatic.

This does not change behavior with outputting of stream data in SCons. If you want anything to help with that, I would recommend look at adding Parts to the SCons build, we have done a bit work to improve logging there.

Jason


-----Original Message-----
From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Dirk Bächle
Sent: Monday, July 7, 2014 3:07 PM
To: SCons users mailing list
Subject: Re: [Scons-users] Announcement: SCons core changes ahead...

Hi Robert,

that was a fast hand raise. ;)

On 07.07.2014 21:38, Robert Zeigler wrote:
> On Jul 7, 2014, at 7/72:10 PM , Dirk Bächle <tshortik at gmx.de> wrote:
>
>> [...]
>>     This change can have an impact on your local code if you're redirecting the SPAWN method in SCons environments your own way.
>>
>>
>> Can you expound on this a bit? I have some situations where I override the SPAWN environment variable to point to a custom method, so I'd like to understand the implications a bit better. In my case, I'm usually kicking off long-running/resource-intensive processes where resource allocation is managed by an external queueing system (SLURM, OGE, etc.).

Not very much, I'm afraid. So far, the integration of the wrapper is in an experimental state only (enough to run first speed tests). The actual authors of the "stubprocess.py" module (Eugene Leskinen, Jason Kenny, et
al.) have tested it to some extent and it's running fine for Parts (the SCons extension). But we might have to change the interface a bit, if problems show up. I also remember having heard about some quirks regarding the returned output (stdout/stderr).
 From what I read between the lines above, you should be pretty safe though.

I hope this give you enough infos to let you sleep well again. ;)

Regards,

Dirk

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
http://four.pairlist.net/mailman/listinfo/scons-users