[Scons-users] Running out of memory while profiling a build. Suggestions?

[Chris BeHanna]

On Dec 26, 2012, at 8:05, Mark Ribau <mribau at realtaentertainment.com> wrote:


> We are attempting to profile our builds, in order to optimize our really really long build times.  We suspect that our custom builders are the particular slow point, but we are unsure which. (We have a number of custom builders, and now that we are fully content populated, our formerly fast builds are very slow. We are hoping that it's not simply that we have lots of content and that we just made some poor decisions in implementing some builders.)

> [...profiling goes OOM...]


William's suggestion to use a 64-bit python can help with OOM.

If you haven't done so already, going through your build systems and refactoring (where possible) any builder calls that have large numbers of sources and targets either into a loop if builder invocations with small numbers of sources and targets, or using the proxy dependency technique described at

http://old.nabble.com/Re%3A-How-to-handle-third-party-modules-that-use-GNU-build-toolchain-within-scons-built-project--p30790357.html

as well as the tips at

http://www.scons.org/wiki/GoFastButton

can provide immediate, dramatic relief. 

Also, if you have cases in which you've managed to insert a directory into your DAG as both a dependency and as a target, and you're modifying that directory's contents during the build, you descend into recursive dependency hell (in scons, a directory always depends upon its children. If those children also depend upon the directory, well, you can see where that will go), which chews up memory and CPU.  Get rid of those if you can. 

-- 
Chris BeHanna
chris at behanna.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://four.pairlist.net/pipermail/scons-users/attachments/20121226/0002a3ff/attachment.html>