[Scons-users] Adding job pools to SCons

[Andrew C. Morrow]

I'd be interested in such a feature, particularly so that it could be
mapped to Ninja. The MongoDB build has implemented something far short of a
pool, that acts as a limiter on the total number of concurrent link jobs.
We needed it because we had a large number of large statically linked unit
tests, and on memory limited systems we could not concurrently link as many
binaries as we had CPUs because we would run out of RAM on some systems.
Having --jlink allowed us to set a higher -j value for compilation, but
constrain the link jobs to fit within memory:

-
https://github.com/mongodb/mongo/blob/d169769bba283cd0a1906c81580ff8e7cf7bcd93/SConstruct#L528-L536
-
https://github.com/mongodb/mongo/blob/d169769bba283cd0a1906c81580ff8e7cf7bcd93/SConstruct#L4465-L4506

It works by creating N dummy targets, and then hashing link targets into N
streams, by declaring the dummy target a side effect. It is definitely
something of a hack, but I thought I would share as it is somewhat
relevant. I have been musing about whether the technique could be
generalized into a pool system or not.


On Fri, May 22, 2020 at 5:05 PM Bill Deegan <bill at baddogconsulting.com>
wrote:

> There was some discussion of similar in users mailing list a while back.
> https://pairlist4.pair.net/pipermail/scons-users/2016-January/004521.html
>
> Yes. Please write up an issue with details..
>
>
> On Fri, May 22, 2020 at 1:13 PM Daniel Moody <dmoody256 at gmail.com> wrote:
>
>> Curious other peoples interest or feedback about adding job pools to
>> SCons?
>>
>> Several other build systems use this idea, which is to group tasks
>> together so that a given group will only utilize a certain number of cpu
>> threads/cores out of the total number of threads/cores originally given to
>> the build. This can improve overall build speed by preventing long jobs
>> from bottle-necking dependencies, or organize jobs which don't run in
>> parallel well (like high IO) from running together.
>>
>> Looking at the Jobs class, I think that class will map pretty well to
>> this concept.
>>
>> I am preparing to write up a gtihub issue up for and wanted to get a
>> census.
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20200523/089d8c2b/attachment.html>