[Scons-users] visualize build parallelism, duration of build steps?

[Gabe Black]

There definitely seems to be a build size component to this. I have a
series of changes which don't change how anything is built, but tell SCons
about everything and then select what to build with different dependencies
instead of using python logic to selectively set up build rules. With -j
set to its previous saturation point performance wise, this actually
increased build time by about 7-8%. I deleted the build and reran to make
sure there weren't caching effects impacting the results, and they stayed
consistent.

Gabe

On Tue, Oct 19, 2021 at 6:59 AM Mats Wichmann <mats at wichmann.us> wrote:

> On 10/18/21 16:45, Gabe Black wrote:
> > Hi folks, sorry if this is a really obvious question, but is there a
> > command line flag or tool or something to visualize how parallel a SCons
> > build is, if there are any bottlenecks, if there are abnormally long
> > running build steps, etc?
>
> The taskmastertrace is generally the way to extract this information.
> It's a bit like trying to drink from a firehose, though, so a tool to
> process it would likely help. Don't know if there are any of those
> floating around.
>
>
> > I've had a suspicion for a few years that there might be a task
> scheduling bug where at certain points a super task seems to come along and
> stop any other tasks from being scheduled until it is done.
> There's been some discussion of this over time.  It appears the way
> scheduling is done may be prone to getting blocked by a long-running
> jobs.  One tuning of the algorithm was proposed in a PR which hasn't
> received much action - opinions about whether this had really pinpointed
> an underlying issue were not unanimous.
>
> https://github.com/SCons/scons/pull/3386
>
> There's also been talk about maybe not having all the members of the
> pool of job threads be handled identically, so there's a place you could
> put a long-running job to get it started as soon as possible, and not
> wait behind a bunch of shorter ones, which could then compete for the
> other threads and maybe get closer to finishing at the same times.
> Making decisions based on length of jobs would require some sort of
> profiling capability so the scheduler has the information available.
>
> On the whole I think it's been posited that there are performance issues
> when the build gets very big, and not sure it's well understood why.
> The taskmaster is a pretty complex piece of code and it was heavily
> tuned "in the old days" (~15 years ago).  Be nice to have better ways to
> characterize what happens on modern hardware configurations with
> potentially larger build projects than when that work was done.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20211019/cb4d1529/attachment-0001.htm>