[Scons-users] Prioritization of builds

[Evan Driscoll]

I have a similar situation to Brian Cody, where if the wrong source 
files get updated, I can wind up with a 30-minute build that does 
*almost everything* in the first 15 minutes but that spends another 15 
minutes serially running a ton of targets. I've /also/ wished that I 
could say "prioritize these targets" so it would build the 
must-be-done-serially targets ASAP and then build everything else around it.

The 'scons <those targets>; scons' script idea wouldn't really help in 
my case; the benefit would be from the fact that much of the first 15 
minutes /could/ be done in parallel with everything from the second 15 
minutes.

The "priority" thing feels pretty natural to me and not anti-SCons... 
you're still not anywhere close to saying "you build this by taking 
these steps." You're still specifying the DAG, you're still letting 
dependency resolution do it's thing, you're just saying "build these 
targets first." I guess something that automatically recorded the time 
and then tried to pick the best scheduling would be a bit clearer, but I 
don't think it's necessary to keep the flavor.


On a somewhat but not entirely unrelated note, the other thing that 
would be sometimes helpful to me is a *memory* aware version of -j. On 
many of our machines, we are memory constrained rather than CPU 
constrained. So the question then comes up with what do you pass as -j? 
If you pass a high number, you start thrashing when it hits the larger 
files; if you pass a small number, there's a lot of lost parallelism 
during much of the build. ('scons <not-memory-constrained targets> -j8; 
scons -j4' /does/ work very well for this problem.)

I've actually thought about and at one point started writing a wrapper 
for gcc that just passes through to real gcc but doesn't invoke the 
compiler until the memory use is below a threshold, but never got around 
to getting everything to work with it. (I don't remember what the 
problem was.) Also thought about making the wrapper silently kill and 
re-invoke the compiler if the memory use started getting too high.

This is another place where, if SCons recorded memory use of the build 
steps, it could use that information to predict a better build order. 
Though I recognize that would be a ton of work.


(I present these things largely as brainstorming; if I can plant the 
idea in the back of your mind or advocate for the priority thing enough 
that it makes it in, that would be helpful I think.)

Evan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160115/16b0894f/attachment.html>