[Scons-users] build time

[Kenny, Jason L]

We have a builds here with about 86K-101K outputs ( with one build taking about 50GB of space in debug config). Before certain tweaks I made in Parts with SCons it was taking about 2-3 minutes to read in about 1000 different part files ( ie these are just SCons files) then to do nothing it would take about 10 minutes.. With work I have done in Parts I got the total time to do nothing case down to about 2 minutes for building everything. And for building smaller sub sets much less. Depending on what and how you want to build it we have startup times down to 13-60 seconds for these massive builds when you don't build everything.

I have looked at a number of different ways to make this faster, and hope to address a few ways in some SCons with to improve the speed and memory of the core SCons logic. I would say as I have been working on this for the last few years that the speed issue is a number of different items, not one simple problem such as the GIL.


Jason

-----Original Message-----
From: scons-users-bounces at scons.org [mailto:scons-users-bounces at scons.org] On Behalf Of TOM TANNER (BLOOMBERG/ LONDON)
Sent: Friday, July 06, 2012 2:43 AM
To: scons-users at scons.org
Subject: Re: [Scons-users] build time

You should have seen how long the build took before I made it non-recursive.

As for size, we have about 25,000 source/header and 13,000 targets files. However all our work is done on NFS drives which does bad things to performance.

I don't think it's the toposort skeleton hitting - I would have thought that would have made the performance worse than the original make system.

I made my comment because I was looking at what scons was doing when it was doing nothing on a multi-cpu system and every thread except one was blocked on the global lock, so we're not making very good use of the available resource.
----- Original Message -----
From: scons-users at scons.org
To: scons-users at scons.org
At:  7/05 02:08:23

On 03 July 2012, TOM TANNER (BLOOMBERG/ LONDON) said:

> I'm converting our build from a.n. other system, and one issue we have 

> is that a 'do nothing' build takes 20 minutes to work out it has to do 

> nothing.


Wow. Just out of curiosity, how many source files do you have? How many build outputs?

(I'm working on rewriting our build with scons, and I'm handling maybe
16,000 of our source files. It's up to 16 sec for a null build, and I'm getting pretty antsy about it. That's about 15 seconds longer than I would like.)


> This is a *little* better on scons, but I notice that even if I give 

> it a lot of threads, it spends all its time hanging around for the 

> infamous global interpreter lock, even though we have a few spare 

> CPUs.


Are you sure it's threading and not the toposort? My understanding is that one of the skeleton's in SCons' closet is that its toposort implementation is O(N^2), whereas a textbook toposort is O(N + E) (number of nodes plus number of edges in the dependency graph).

       Greg
-- 
Greg Ward                                http://www.gerg.ca/
No animals were harmed in transmitting this message.
_______________________________________________
Scons-users mailing list
Scons-users at scons.org
http://four.pairlist.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
http://four.pairlist.net/mailman/listinfo/scons-users