[Scons-users] visualize build parallelism, duration of build steps?

[Gabe Black]

To follow up with some measurements, I found that my build performance
scales logarithmically with -j, up to the number of threads in my system. I
have a 12 core CPU with 2 threads per core, so that is up to 24 threads.
After that point, performance stops improving and actually degrades
slightly. By logarithmically, I mean that the graph of the log of the -j
value tracks the decrease in run time of a clean build reasonably well. Or
in other words, I would expect going from -j4 to -j16 to improve build time
by a factor of 2x, and not a factor of 4x.

Is this expected? Is there a way to diagnose what keeping our build from
scaling roughly linearly? We have many individual object files which should
be buildable in parallel, so I'm not sure what the bottleneck is. There
curve seems to be fairly smooth and not a hard drop off, so I don't think
we're hitting a specific limit in memory bandwidth or capacity, overflowing
the disk cache, etc.

Gabe

On Mon, Oct 18, 2021 at 3:45 PM Gabe Black <gabe.black at gmail.com> wrote:

> Hi folks, sorry if this is a really obvious question, but is there a
> command line flag or tool or something to visualize how parallel a SCons
> build is, if there are any bottlenecks, if there are abnormally long
> running build steps, etc?
>
> I don't have an immediate pressing need for a tool like that, but I think
> it might be helpful identifying places our build could be tightened up. It
> could and also help get a rough idea of how well the build would scale to a
> very parallel build machine, ie. what might build time be with 32 cores? 64
> cores? 128 cores? etc. How much memory would you need for 128 cores to
> avoid memory bottlenecks?
>
> Somewhat on that topic, are there any relatively easy to set up
> distributed build mechanisms for SCons? I'm sure you could plug distcc into
> it for instance, but is there anything which has some nice, turn key(ish?)
> integration?
>
> Thanks!
> Gabe
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20211019/fbe98571/attachment.htm>