[Scons-users] build time

[TOM TANNER (BLOOMBERG/ LONDON)]

That would be good.

I'm not sure why it'd have to require 2.6 (as someone suggested in another mail), as presumably it could use the same system of determining whether multiprocessing is available as is used to determine whether or not threading is available, by doing
try:
    import multiprocessing
    -- multiprocessing version of parallel
except NameError
    -- threading version of parallel


----- Original Message -----
From: scons-users at scons.org
To: scons-users at scons.org
At:  7/07 11:08:06

What would be cool would be for SCons to be able to use the Python
multiprocessing module instead of threads. The API is very similar by
design, so it shouldnt be too difficult to change. Then all available
CPUs/cores could be used without having to deal with the GIL. And maybe
this would be too much, but if it were configurable, that would just be
awesome.

Thoughts, anyone?

Brady

On Fri, Jul 6, 2012 at 7:39 PM, William Deegan <bill at baddogconsulting.com>wrote:


> Tom,

>

> On Jul 6, 2012, at 12:43 AM, TOM TANNER (BLOOMBERG/ LONDON) wrote:

>

> > You should have seen how long the build took before I made it

> non-recursive.

> >

> > As for size, we have about 25,000 source/header and 13,000 targets

> files. However all our work is done on NFS drives which does bad things to

> performance.

> >

> > I don't think it's the toposort skeleton hitting - I would have thought

> that would have made the performance worse than the original make system.

> >

> > I made my comment because I was looking at what scons was doing when it

> was doing nothing on a multi-cpu system and every thread except one was

> blocked on the global lock, so we're not making very good use of the

> available resource.

>

> Do you have some custom builders which are implemented in python code in

> your SConstruct/SConscript?

> -Bill

>

>

> > ----- Original Message -----

> > From: scons-users at scons.org

> > To: scons-users at scons.org

> > At:  7/05 02:08:23

> >

> > On 03 July 2012, TOM TANNER (BLOOMBERG/ LONDON) said:

> >> I'm converting our build from a.n. other system, and one issue we

> >> have is that a 'do nothing' build takes 20 minutes to work out it has

> >> to do nothing.

> >

> > Wow. Just out of curiosity, how many source files do you have? How

> > many build outputs?

> >

> > (I'm working on rewriting our build with scons, and I'm handling maybe

> > 16,000 of our source files. It's up to 16 sec for a null build, and

> > I'm getting pretty antsy about it. That's about 15 seconds longer than

> > I would like.)

> >

> >> This is a *little* better on scons, but I notice that even if I give

> >> it a lot of threads, it spends all its time hanging around for the

> >> infamous global interpreter lock, even though we have a few spare

> >> CPUs.

> >

> > Are you sure it's threading and not the toposort? My understanding is

> > that one of the skeleton's in SCons' closet is that its toposort

> > implementation is O(N^2), whereas a textbook toposort is O(N + E)

> > (number of nodes plus number of edges in the dependency graph).

> >

> >       Greg

> > --

> > Greg Ward                                http://www.gerg.ca/

> > No animals were harmed in transmitting this message.

> > _______________________________________________

> > Scons-users mailing list

> > Scons-users at scons.org

> > http://four.pairlist.net/mailman/listinfo/scons-users

> >

> > _______________________________________________

> > Scons-users mailing list

> > Scons-users at scons.org

> > http://four.pairlist.net/mailman/listinfo/scons-users

>

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://four.pairlist.net/pipermail/scons-users/attachments/20120709/04fc5e4e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 12412913325782002.txt
Type: application/octet-stream
Size: 149 bytes
Desc: not available
Url : <http://four.pairlist.net/pipermail/scons-users/attachments/20120709/04fc5e4e/attachment.obj>