[Scons-users] Constraining the number of concurrent link jobs

[Andrew C. Morrow]

I stumbled on an interesting technique with SCons that I thought was worth
sharing. I can't call the solution exactly pretty, but it does seem to work.

The motivating issue is that in the codebase I work on we have a large
number of static test binaries in our CI build. On some systems, the linker
is very memory hungry, and linking these static test binaries consumes a
large amount of memory. So much so that we can't have more than two or
three executions of the linker running concurrently or we will run the
system out of memory and the build will fail.

As a result, we end up building with -j2 or -j3, even though the machine in
question has far more cores. That means that the compile phase can't
capitalize on the available parallelism, slowing down the overall build.
What we would like would be to have something more like Ninja's "pools",
where we can build with -j16, but then restrict things so that no more than
2 or 3 link jobs can run concurrently.

We achieved that by adding an Emitter to each of the Program and
SharedLibrary nodes. The emitter declares that members of the target array
all share a dummy SideEffect file, where the dummy filename is chosen by
hashing the node name of the first target and then modding the hash by the
desired maximum link parallelism. The result is that no more than the
selected number of linker executions can run concurrently, since targets
that share a SideEffect will never run concurrently. A working but rough
implementation can be seen here:

https://github.com/mongodb/mongo/blob/2edb91e2a51f0bdcf89f51b1a3281443c13e52c5/SConstruct#L3497-L3531

For our implementation, we only cared about constraining the number of link
jobs, but the technique is general enough that you could have an arbitrary
number of different job pools.

Like I said, not very pretty, but it has the advantage that it requires no
modifications to SCons. There is probably a better way to do this at the
taskmaster level, and maybe such a feature should be considered for a
future version of SCons. In the meantime, I hope the above trick proves
useful to someone.

Thanks,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20180512/23506341/attachment.html>