[Scons-users] build time

[Brady Johnson]

ok, thanks. I didnt get a chance to answer yesterday regarding using the
Python multiprocessing module. Those were excellent points brought up about
sharing the data, etc.

Regards,

Brady

On Mon, Jul 9, 2012 at 7:52 PM, William Deegan <bill at baddogconsulting.com>wrote:


> Brady,

> On Jul 9, 2012, at 12:28 AM, Brady Johnson wrote:

>

> Bill,

>

> Sorry for not answering, I thought the question was directed to Tom

> Tanner. Personally, I dont have many builders with involved Python logic.

> Looking at it from the point of view you just mentioned, it makes sense

> that the GIL shouldnt have that much impact, but I guess it would be better

> to test it.

>

>

>

> If you move any python based builders to scripts which can be invoked you

> will see your parallelism go up.

> I've made this mistake in the past.

> In general on larges builds I'll see all my cpu's pegged by using -J from

> start to end of the build barring dependencies as described before with

> dependency trees which get wide, and then narrow..

>

> -Bill

>

>

> Regards,

>

> Brady

>

> On Mon, Jul 9, 2012 at 12:21 AM, William Deegan <bill at baddogconsulting.com

> > wrote:

>

>> Russel,

>>

>> On Jul 7, 2012, at 4:30 AM, Russel Winder wrote:

>>

>> > On Sat, 2012-07-07 at 12:07 +0200, Brady Johnson wrote:

>> >> What would be cool would be for SCons to be able to use the Python

>> >> multiprocessing module instead of threads. The API is very similar by

>> >> design, so it shouldnt be too difficult to change. Then all available

>> >> CPUs/cores could be used without having to deal with the GIL. And maybe

>> >> this would be too much, but if it were configurable, that would just be

>> >> awesome.

>> >>

>> >> Thoughts, anyone?

>> >

>> > This would require setting the SCons Python floor to 2.6 – well it could

>> > be done with 2.5 as there is a backport of multiprocessing to 2.5, but I

>> > think 2.6 would be a better option.  The problem is that there are a lot

>> > of 2.4 users out there who are unwilling and/or unable to upgrade.

>> >

>> > There are many options.

>> >

>> > 1. Up the floor anyway and only allow SCons updates to be used by people

>> > runing 2.6 or higher.

>> >

>> > 2. Do nothing and continue to use 2.4 as the floor.

>> >

>> > 3.  Use the PyPy with STM instead of a GIL.

>> >

>> > 4. Ensure all actions get spawned via ctypes of some such so they are

>> > not running Python bytecodes and thus not constrained by GIL.

>> >

>> > 5. Rewrite SCons in Python 3 an make some real progress whilst really

>> > annoying people stuck on Python 2.4 :-)

>>

>> As I understand it, the GIL only becomes a problem for SCons if you have

>> builders which are python logic. If you change such that they are scripts

>> run from the command line, then the GIL no longer comes into play.

>>

>> Also important to keep in mind that depending on the specifics of a build

>> there may be only so much parallelism which would be possible.

>> (Think 100 sources files each build 2 libraries which then build a

>> program, which then is run to generate source files...)

>>

>> Thus my question (unanswered) to Brady if he had any python builder

>> logic...

>>

>> -Bill

>> _______________________________________________

>> Scons-users mailing list

>> Scons-users at scons.org

>> http://four.pairlist.net/mailman/listinfo/scons-users

>>

>

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

>

>

>

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

>

>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://four.pairlist.net/pipermail/scons-users/attachments/20120710/b953faab/attachment.htm>