[Scons-users] Not able to make full use of CPUs for processing speed

[Don Baldwin]

Our value of N is much less than 100.  We’ll look into the MongoDB solution.  Thanks.

-Don

From: Scons-users <scons-users-bounces at scons.org> On Behalf Of Bill Deegan
Sent: Wednesday, September 15, 2021 4:09 PM
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] Not able to make full use of CPUs for processing speed


WARNING: This email originated from outside of Qualcomm. Please be wary of any links or attachments, and do not enable macros.
Don,

How big is your N?
For reasonable N's SCons can keep up, but for N's in the 100s it' has some issues.
The folks a MongoDB are using icecc and have built with N's > 250.. and SCons has issues.
They've floated a PR to address a shortcoming in the scheduler which helped their build, but it's still marked a WIP so not merged.
see: https://github.com/SCons/scons/pull/3386

You could try making those changes locally and see if that helps.

-Bill


On Wed, Sep 15, 2021 at 3:45 PM Don Baldwin <donb at qti.qualcomm.com<mailto:donb at qti.qualcomm.com>> wrote:
Hi,

We have a build machine with, say, N CPUs, and we try increasing num_jobs to a higher value, but we’re not seeing a decrease in build times.

For example, say we have 10 CPUs.  We would expect that as num_jobs approaches 10, we would continue to see better and better performance during the compile phase.  However, what we’re seeing is that performance improves until num_jobs hits about 5, at which point further increases to num_jobs isn’t having an effect.

My suspicion is that SCons is doing a lot of processing on it’s own trying to figure out which jobs to queue next, e.g. scanning .c files for new implicit dependencies, and that there’s a theoretical limit on how many CPUs it’s able to keep up with.  Could something like that be going on?

Thanks,
Don
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210916/b00c5e71/attachment-0001.htm>