[Scons-users] build time

[Kenny, Jason L]

I should add that technically multiprocessing has few huge issues with being used in SCons.

1) it is not good for systems that need to share lots of data. As it turns out SCons has this node tree that has to be shared. This would in turn show down SCon and increase memory usage like crazy.
2) it requires the shared objects to be pickable. As the structures are currently in SCons this is not the case, and if it was it would currently pickle the whole most of the memory as the node and environment and node tree are linked in a not so good way at the moment
This for me is enough... the other issue 3) is that the we need to pass pickled functions, or we have to re-read all the data files for each process we spawn. The other options is that we keep the pure python based builders as is and run then in a thread pool, and hope the imple of python has the GIL issue fixed. Given this is the current approach, and that technically only command line actions can run well with multiprocessing, there is no advantage to using it at the moment as the current thread level is enough for these cases.

To add to the below.. I would also add that SCons could prevent python based builders.. and instead make it easy to write a python builder that can be spawned as a separate script.

Jason

-----Original Message-----
From: scons-users-bounces at scons.org [mailto:scons-users-bounces at scons.org] On Behalf Of Russel Winder
Sent: Saturday, July 07, 2012 6:31 AM
To: SCons users mailing list
Subject: Re: [Scons-users] build time

On Sat, 2012-07-07 at 12:07 +0200, Brady Johnson wrote:

> What would be cool would be for SCons to be able to use the Python 

> multiprocessing module instead of threads. The API is very similar by 

> design, so it shouldnt be too difficult to change. Then all available 

> CPUs/cores could be used without having to deal with the GIL. And 

> maybe this would be too much, but if it were configurable, that would 

> just be awesome.

> 

> Thoughts, anyone?


This would require setting the SCons Python floor to 2.6 – well it could be done with 2.5 as there is a backport of multiprocessing to 2.5, but I think 2.6 would be a better option.  The problem is that there are a lot of 2.4 users out there who are unwilling and/or unable to upgrade.

There are many options.

1. Up the floor anyway and only allow SCons updates to be used by people runing 2.6 or higher.

2. Do nothing and continue to use 2.4 as the floor.

3.  Use the PyPy with STM instead of a GIL.

4. Ensure all actions get spawned via ctypes of some such so they are not running Python bytecodes and thus not constrained by GIL.

5. Rewrite SCons in Python 3 an make some real progress whilst really annoying people stuck on Python 2.4 :-)

--
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder