[Scons-users] Not able to make full use of CPUs for processing speed

[Bill Deegan]

you can also use the --taskmastertrace=FILE scons option and look at the
output file, that may give you hints why you're not getting the parallelism
you suspect, though that's a deeper dive.

On Thu, Sep 16, 2021 at 7:31 AM Don Baldwin <donb at qti.qualcomm.com> wrote:

> Our value of N is much less than 100.  We’ll look into the MongoDB
> solution.  Thanks.
>
>
>
> -Don
>
>
>
> *From:* Scons-users <scons-users-bounces at scons.org> * On Behalf Of *Bill
> Deegan
> *Sent:* Wednesday, September 15, 2021 4:09 PM
> *To:* SCons users mailing list <scons-users at scons.org>
> *Subject:* Re: [Scons-users] Not able to make full use of CPUs for
> processing speed
>
>
>
> *WARNING:* This email originated from outside of Qualcomm. Please be wary
> of any links or attachments, and do not enable macros.
>
> Don,
>
>
>
> How big is your N?
>
> For reasonable N's SCons can keep up, but for N's in the 100s it' has some
> issues.
>
> The folks a MongoDB are using icecc and have built with N's > 250.. and
> SCons has issues.
>
> They've floated a PR to address a shortcoming in the scheduler which
> helped their build, but it's still marked a WIP so not merged.
>
> see: https://github.com/SCons/scons/pull/3386
>
>
>
> You could try making those changes locally and see if that helps.
>
>
>
> -Bill
>
>
>
>
>
> On Wed, Sep 15, 2021 at 3:45 PM Don Baldwin <donb at qti.qualcomm.com> wrote:
>
> Hi,
>
>
>
> We have a build machine with, say, N CPUs, and we try increasing num_jobs
> to a higher value, but we’re not seeing a decrease in build times.
>
>
>
> For example, say we have 10 CPUs.  We would expect that as num_jobs
> approaches 10, we would continue to see better and better performance
> during the compile phase.  However, what we’re seeing is that performance
> improves until num_jobs hits about 5, at which point further increases to
> num_jobs isn’t having an effect.
>
>
>
> My suspicion is that SCons is doing a lot of processing on it’s own trying
> to figure out which jobs to queue next, e.g. scanning .c files for new
> implicit dependencies, and that there’s a theoretical limit on how many
> CPUs it’s able to keep up with.  Could something like that be going on?
>
>
>
> Thanks,
>
> Don
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210916/453e55c4/attachment-0001.htm>