[Scons-users] Need help to improve the incremental build time while using SCons

[Jason Kenny]

I got around the issue with build speeds with work I did in Parts. This breaks up the build files in to components and applies MD5 checksomes to the files. The system then load build files that have not been built or have changes in then. This greatly reduced build time for very large projects. String subst() code has an effect, but the main issue is that load time to get started, which for build I had in the past could take 20 minutes to start with all the files being loaded and scanning of the disk, etc… The code I had could take turn that build time for a up-to-date build down to 14-20 seconds.

Jason

From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Thomas Berg
Sent: Tuesday, June 20, 2017 3:59 AM
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] Need help to improve the incremental build time while using SCons

On Tue, Jun 20, 2017 at 7:29 AM, Basil Mathew <iambasilmathew at gmail.com<mailto:iambasilmathew at gmail.com>> wrote:
1. Is the nearly 6 fold increase in the incremental build time as compared to GMake on expected lines? I did go through the archive e-mails, but could not get a concrete answer to the question.

Yes, this is my experience with large C++ projects too. With ~9000 .cpp files, we get a no-op build time of about 5 minutes after implementing many of the optimisations from the mailing list.


2. Are there any other options that would allow me to get a more favorable incremental build time?

In our build, tests have shown that we could cut off minutes by not scanning third party headers. But we update our libraries sufficiently often that it's not something we can use by default.

SCons is also known to spend a lot of time doing string substitutions, generating compiler commands. You may get some speedup from modifying the compiler command, hard coding as much as possible in the command itself instead of adding things to CCFLAGS and CPPDEFINES. You can do this for the parts that are the same for your entire build.

The biggest win for incremental builds is to support partial builds: in our build we detect it if a developer launches scons in a subdirectory, and in this case we execute the SConscripts for this subtree only, meaning we don't inform scons about the rest of the build targets. You may need to modify your SConscripts and even implement workarounds for scons bugs (in some cases) to support this. And of course developers then have to know what they are doing to get a correct build - they have to know that their modification only requires building a particular subtree. But we get 10 second incremental build times from this, for single .cpp file changes (and headers too, assuming they are only included in the subtree in question), so for us this is an absolute requirement.

Cheers,
Thomas Berg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170620/2b9b1686/attachment-0001.html>