[Scons-users] Speeding up subprocess spawns

[Yonatan]

Hello, SCons community,

Recently, I debugged the build-process performance of a large project built
with SCons.
Trying to figure out exactly where time is spent (gcc vs. scons vs.
linking), I ran a system-wide "perf".
One of the things that surprised me was that Python spends a significant
amount of time in fork() and
execve() (in the kernel).

Almost all of the time of fork() was spent in copy_page_range(), and of
execve() in unmap_page_range().
To my understanding, these 2 kernel functions are responsible to copy the
process' page tables when
fork()ing, and to remove existing mappings when execve()ing (as you switch
to a new address space with
new mappings). The larger resident set a process has, the more time those 2
functions will take.

In that project, the SCons process grows very large in RAM (over 1GB); this
is obviously specific to that
project, and what it loads into Python. With a resident set that large, the
fork()-execve() sequence incurs a
significant cost.

To overcome this I thought of 2 possible solutions:
1. Use vfork()-execve() in subprocess.Popen() - this is not supported in
yet CPython, but it has been
    suggested and a PR exists. See https://bugs.python.org/issue35823
2. Run a minimal "spawner" process and execute subprocess.Popen() requests
via it. I've written a PoC, and
    tested MongoDB's compilation using it. SCons doesn't hog too much
CPU/memory in the MongoDB case,
    but the spawner change still introduces a tiny improvement.
    I also tried manually increasing the memory usage of SCons (by creating
many Python objects), which
    lead to much more visible effects for this change.
    You can see it here: https://github.com/Jongy/scons/tree/process-spawner

I hope the vfork() PR gets merged, and meanwhile I was wondering if the
"spawner" approach sounds feasible to you.
I'd be happy to push it towards mainlining.

Thanks,
Yonatan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20200625/11d6ebac/attachment.html>