[Scons-users] scons cache in a CI scenario

[Mats Wichmann]

On 10/29/18 11:19 AM, Andrew C. Morrow wrote:
> My company has done this by using Amazon EFS <https://aws.amazon.com/efs/>
> to back the SCons cache for the parts of our CI system that run in AWS, and
> the results have been excellent.
> 
> We create a new cache directory on EFS for each new build image version
> that we run, so that changes to the build image that may not be visible to
> SCons are not a concern. We discussed at one point making it so that the
> SCons cache could be backed by something other than a filesystem, but EFS
> served well enough for us that we never pursued that further. It is also
> important to consider how you will prune the cache, or identify old caches
> that are no longer in active use, so that the EFS footprint doesn't grow
> without bound. We wrote scripts for that which evaluate the most recent
> access time of the files (and we had to make small modification to SCons to
> get the timestamps right) and only retain the most recently accessed files
> in the cache.
> 
> Overall, It is definitely a project worth pursuing to reduce CI build
> times: it isn't trivial to set up, but the build time savings were for us
> very real.
> 
> Thanks,
> Andrew

I didn't respond and meant to: thanks for this info!  I believe in our
case (I checked with the host of our infra) it's possible to set
something up in OpenStack that will appear as part of the filesystem of
the instances.

I got some surprising pushback when bringing up this possibility in the
project - I brought it up because frankly I'm tired of spending two
hours or more waiting for an answer from the CI system... something that
isn't really important to the end product, but is important to the
developer experience. I'll just paste the relevant bits instead of
paraphrasing:

=========
a cached build is not a good idea.

Number one reason for not using a cached build. "Reliability".
  - Case 1: A problem is solved just by doing a clean build.
  - Case 2: A problem is hidden because a left over build product from
the cached build items.

A false failure like Case 1 where the build fails but can be fixed with
a clean build is annoying but not a critical problem as long as there is
a way to force a clean build. The big problem is Case 2 where it passes
because of picking up something from the cached build. The error will,
likely, not be caught till it is merged. Both problems I agree are
unusual but they still happen.

I am all for speeding up the build but I suspect adding cached build
would cause more problems than its worth.
=========

I still don't fully grok how the various signatures are generated, which
I presume is the answer to the reliability question.  I presume it would
be really hard to get a "wrong answer" in the scons scenario?

I have also worried about what Andrew mentioned - not as a "stopper" but
just something that needs doing - how do you keep the cache fresh and
not just growing over time with eventually unneeded artifacts, etc.

Other (cough) builsystems seem to have paid a lot of attention to
sharing build artifacts through caching.  In particular:

https://docs.bazel.build/versions/master/remote-caching.html

Scons does not seem to have any concept of a networked cache - it would
work if you had the network connected to the local filesystem or the
schemes already mentioned in this thread, but to the viewpoint of SCons
that's all local.  Does thinking about a network cache make any sense?
I'd worry that time-deciding-if-we're-going-to-use followed by
time-actually-fetching may often be more than just building locally,
esp. if you follow a programming model that leaves you with relatively
small source files.  I'm happy to stop thinking about it if we know it's
not a particularly profitable model :)

"Problem solved by doing a clean build" - I presume that would be giving
the --cache-disable flag, since otherwise the files "come back" after a
clean, right?