[Scons-users] build time

[William Deegan]

Russel,

On Jul 7, 2012, at 4:30 AM, Russel Winder wrote:


> On Sat, 2012-07-07 at 12:07 +0200, Brady Johnson wrote:

>> What would be cool would be for SCons to be able to use the Python

>> multiprocessing module instead of threads. The API is very similar by

>> design, so it shouldnt be too difficult to change. Then all available

>> CPUs/cores could be used without having to deal with the GIL. And maybe

>> this would be too much, but if it were configurable, that would just be

>> awesome.

>> 

>> Thoughts, anyone?

> 

> This would require setting the SCons Python floor to 2.6 – well it could

> be done with 2.5 as there is a backport of multiprocessing to 2.5, but I

> think 2.6 would be a better option.  The problem is that there are a lot

> of 2.4 users out there who are unwilling and/or unable to upgrade.

> 

> There are many options.

> 

> 1. Up the floor anyway and only allow SCons updates to be used by people

> runing 2.6 or higher.

> 

> 2. Do nothing and continue to use 2.4 as the floor.

> 

> 3.  Use the PyPy with STM instead of a GIL.

> 

> 4. Ensure all actions get spawned via ctypes of some such so they are

> not running Python bytecodes and thus not constrained by GIL.

> 

> 5. Rewrite SCons in Python 3 an make some real progress whilst really

> annoying people stuck on Python 2.4 :-)


As I understand it, the GIL only becomes a problem for SCons if you have builders which are python logic. If you change such that they are scripts run from the command line, then the GIL no longer comes into play.

Also important to keep in mind that depending on the specifics of a build there may be only so much parallelism which would be possible.
(Think 100 sources files each build 2 libraries which then build a program, which then is run to generate source files...)

Thus my question (unanswered) to Brady if he had any python builder logic...

-Bill