[Scons-users] unit testing best practices and parallel builds

[Mats Wichmann]

There's a fair bit of stuff on this topic on the internet, I've done
some of searching and reading, but I thought I'd come here for a few
opinions maybe.

The project in question is fairly large size (a push to Coverity just
recently resulted in a claim of 700k lines of code). It has unit tests,
not nearly as many as it should, but a decent number in the area that is
generally considered "the API". Because of the size, the build is not
speedy, and the project is wired up to a Continuous Integration
framework that builds a whole bunch of combinations of build options and
host/target platforms, which collectively are used to generate a +1 or
-1 on each proposed commit from the point of view of "doesn't break the
build". (There are like a dozen and a half distinct builds now, and it's
only likely to grow). That, naturally, is a problem if it takes a long
time to get an answer when requests pile on top of requests, so builds
default to being as parallel as possible. In fact, the script that the
autobuilders invoke do this, and since the host has configured the
various virtual machines "optimistically", as in told they're 16
processors, so, they're Quite Parallel:

if os.getenv("SCONSFLAGS", "") == "":
    os.environ["SCONSFLAGS"] = "-Q -j " + str(multiprocessing.cpu_count())

If a build runs unit tests it sometimes trips over itself for no
apparent reason ("transient" - though there's likely some underlying
problem to solve), and because of the parallelism, it's sometimes quite
hard to sort out in the logfile amongst the interleaving messages what's
happening.

So with all of that intro...

are there Best Practices for making something that isn't really part of
the "build" but is done at build time - namely running unit tests -
happen separately at the end of the chain so they're not in the middle
of the parallel build?