[Scons-users] scons cache in a CI scenario

[Andrew C. Morrow]

Thanks for writing back. A few quick things to consider that may help sway
the argument:

- The whole point of using SCons is that its builds are supposed to be 100%
reproducible. If you have a non-reproducible build, then you have a flaw in
your implementation of SCons (or there is an SCons bug). The most common
sort of flaw is failing to declare dependencies. This is especially common
in uses of Command.

- However, the concerns about incorrect builds are fair. Bugs, whether in
SCons or your deployment of it, are a fact of life. One mitigation is to
only use caching for builds that will not go out the door. Our release
builds do not interact with the cache at all. Only developer "patch" builds
and regular commits do. In practice, we have not encountered any adverse
consequences of using the cache, but release bits are simply to precious to
chance it.

- Along with cache pruning work, we have implemented a global kill switch
for the cache, so that if something does go badly wrong, we can quickly
disable it everywhere it is used. We have yet to use it.

- Direct network support for CacheDir in SCons is probably a good idea,
especially if there is skepticism about the semantics of NFS or CIFS. The
first step would be to make the cache dir implementation pluggable and
abstract away the FS operations. Then write a different storage backend for
it (S3?).

- Pruning the cache correctly does take some thought. One way to avoid
bloating the cache is to avoid putting statically linked artifacts into it,
as these tend to be large. Dynamic libraries or things linked to dynamic
libraries should be fine. It is easy to add an optional emitter to disable
the cache for certain types of targets
<https://github.com/mongodb/mongo/blob/master/SConstruct#L1271-L1290>.

- Performance is of course a concern. You need to do your own research and
experiments to determine if it is a win in your build.

Overall, you need to treat moving to a networked/shared cache as a project,
not just something to turn on. Do a proper design, build consensus, have an
ops plan, etc.

Here is a pretty picture of what happened to our CI compile times after
deploying the EFS backed cache:

[image: image.png]

Hope that helps.

Thanks,
Andrew



On Wed, Oct 31, 2018 at 11:35 AM Mats Wichmann <mats at wichmann.us> wrote:

> On 10/29/18 11:19 AM, Andrew C. Morrow wrote:
> > My company has done this by using Amazon EFS <
> https://aws.amazon.com/efs/>
> > to back the SCons cache for the parts of our CI system that run in AWS,
> and
> > the results have been excellent.
> >
> > We create a new cache directory on EFS for each new build image version
> > that we run, so that changes to the build image that may not be visible
> to
> > SCons are not a concern. We discussed at one point making it so that the
> > SCons cache could be backed by something other than a filesystem, but EFS
> > served well enough for us that we never pursued that further. It is also
> > important to consider how you will prune the cache, or identify old
> caches
> > that are no longer in active use, so that the EFS footprint doesn't grow
> > without bound. We wrote scripts for that which evaluate the most recent
> > access time of the files (and we had to make small modification to SCons
> to
> > get the timestamps right) and only retain the most recently accessed
> files
> > in the cache.
> >
> > Overall, It is definitely a project worth pursuing to reduce CI build
> > times: it isn't trivial to set up, but the build time savings were for us
> > very real.
> >
> > Thanks,
> > Andrew
>
> I didn't respond and meant to: thanks for this info!  I believe in our
> case (I checked with the host of our infra) it's possible to set
> something up in OpenStack that will appear as part of the filesystem of
> the instances.
>
> I got some surprising pushback when bringing up this possibility in the
> project - I brought it up because frankly I'm tired of spending two
> hours or more waiting for an answer from the CI system... something that
> isn't really important to the end product, but is important to the
> developer experience. I'll just paste the relevant bits instead of
> paraphrasing:
>
> =========
> a cached build is not a good idea.
>
> Number one reason for not using a cached build. "Reliability".
>   - Case 1: A problem is solved just by doing a clean build.
>   - Case 2: A problem is hidden because a left over build product from
> the cached build items.
>
> A false failure like Case 1 where the build fails but can be fixed with
> a clean build is annoying but not a critical problem as long as there is
> a way to force a clean build. The big problem is Case 2 where it passes
> because of picking up something from the cached build. The error will,
> likely, not be caught till it is merged. Both problems I agree are
> unusual but they still happen.
>
> I am all for speeding up the build but I suspect adding cached build
> would cause more problems than its worth.
> =========
>
> I still don't fully grok how the various signatures are generated, which
> I presume is the answer to the reliability question.  I presume it would
> be really hard to get a "wrong answer" in the scons scenario?
>
> I have also worried about what Andrew mentioned - not as a "stopper" but
> just something that needs doing - how do you keep the cache fresh and
> not just growing over time with eventually unneeded artifacts, etc.
>
> Other (cough) builsystems seem to have paid a lot of attention to
> sharing build artifacts through caching.  In particular:
>
> https://docs.bazel.build/versions/master/remote-caching.html
>
> Scons does not seem to have any concept of a networked cache - it would
> work if you had the network connected to the local filesystem or the
> schemes already mentioned in this thread, but to the viewpoint of SCons
> that's all local.  Does thinking about a network cache make any sense?
> I'd worry that time-deciding-if-we're-going-to-use followed by
> time-actually-fetching may often be more than just building locally,
> esp. if you follow a programming model that leaves you with relatively
> small source files.  I'm happy to stop thinking about it if we know it's
> not a particularly profitable model :)
>
> "Problem solved by doing a clean build" - I presume that would be giving
> the --cache-disable flag, since otherwise the files "come back" after a
> clean, right?
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20181031/612e5743/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 442314 bytes
Desc: not available
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20181031/612e5743/attachment-0001.png>