[Scons-users] build time

William Deegan bill at baddogconsulting.com
Mon Jul 9 13:52:32 EDT 2012


Brady,
On Jul 9, 2012, at 12:28 AM, Brady Johnson wrote:


> Bill,

>

> Sorry for not answering, I thought the question was directed to Tom Tanner. Personally, I dont have many builders with involved Python logic. Looking at it from the point of view you just mentioned, it makes sense that the GIL shouldnt have that much impact, but I guess it would be better to test it.



If you move any python based builders to scripts which can be invoked you will see your parallelism go up.
I've made this mistake in the past.
In general on larges builds I'll see all my cpu's pegged by using -J from start to end of the build barring dependencies as described before with dependency trees which get wide, and then narrow..

-Bill


>

> Regards,

>

> Brady

>

> On Mon, Jul 9, 2012 at 12:21 AM, William Deegan <bill at baddogconsulting.com> wrote:

> Russel,

>

> On Jul 7, 2012, at 4:30 AM, Russel Winder wrote:

>

> > On Sat, 2012-07-07 at 12:07 +0200, Brady Johnson wrote:

> >> What would be cool would be for SCons to be able to use the Python

> >> multiprocessing module instead of threads. The API is very similar by

> >> design, so it shouldnt be too difficult to change. Then all available

> >> CPUs/cores could be used without having to deal with the GIL. And maybe

> >> this would be too much, but if it were configurable, that would just be

> >> awesome.

> >>

> >> Thoughts, anyone?

> >

> > This would require setting the SCons Python floor to 2.6 – well it could

> > be done with 2.5 as there is a backport of multiprocessing to 2.5, but I

> > think 2.6 would be a better option. The problem is that there are a lot

> > of 2.4 users out there who are unwilling and/or unable to upgrade.

> >

> > There are many options.

> >

> > 1. Up the floor anyway and only allow SCons updates to be used by people

> > runing 2.6 or higher.

> >

> > 2. Do nothing and continue to use 2.4 as the floor.

> >

> > 3. Use the PyPy with STM instead of a GIL.

> >

> > 4. Ensure all actions get spawned via ctypes of some such so they are

> > not running Python bytecodes and thus not constrained by GIL.

> >

> > 5. Rewrite SCons in Python 3 an make some real progress whilst really

> > annoying people stuck on Python 2.4 :-)

>

> As I understand it, the GIL only becomes a problem for SCons if you have builders which are python logic. If you change such that they are scripts run from the command line, then the GIL no longer comes into play.

>

> Also important to keep in mind that depending on the specifics of a build there may be only so much parallelism which would be possible.

> (Think 100 sources files each build 2 libraries which then build a program, which then is run to generate source files...)

>

> Thus my question (unanswered) to Brady if he had any python builder logic...

>

> -Bill

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

>

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://four.pairlist.net/pipermail/scons-users/attachments/20120709/0af33efb/attachment-0001.html>


More information about the Scons-users mailing list