[Scons-users] scons cache in a CI scenario

[Andrew C. Morrow]

My company has done this by using Amazon EFS <https://aws.amazon.com/efs/>
to back the SCons cache for the parts of our CI system that run in AWS, and
the results have been excellent.

We create a new cache directory on EFS for each new build image version
that we run, so that changes to the build image that may not be visible to
SCons are not a concern. We discussed at one point making it so that the
SCons cache could be backed by something other than a filesystem, but EFS
served well enough for us that we never pursued that further. It is also
important to consider how you will prune the cache, or identify old caches
that are no longer in active use, so that the EFS footprint doesn't grow
without bound. We wrote scripts for that which evaluate the most recent
access time of the files (and we had to make small modification to SCons to
get the timestamps right) and only retain the most recently accessed files
in the cache.

Overall, It is definitely a project worth pursuing to reduce CI build
times: it isn't trivial to set up, but the build time savings were for us
very real.

Thanks,
Andrew



On Mon, Oct 29, 2018 at 1:05 PM Mats Wichmann <mats at wichmann.us> wrote:

>
> Recent discussions about the scons cache, which I've never made use of
> in the project that brought me over here to scons, got me to thinking...
>
> A continuous integration setup could potentially benefit from the cache
> - if there was a way to share it appropriately.
>
> As far as I can see, a cache is a local entity - you give it a directory
> name, which could potentially be shared (NFS, etc.), but still depends
> on having a standard pathname.
>
> For builds in a CI system, instances of the builder are constantly spun
> up, used, and discarded, so you don't have persistence - in fact lack of
> persistence is usually considered a benefit, just like scons's own
> reproducible build concept of ignoring the shell environment.
>
> Our builders - there are a bunch, for several different target systems,
> do very similar things over and over typically with only very small
> changes - you submit a patch to our Gerrit instance, it kicks off a
> dozen builds, something goes wrong, you find the bug in one file and
> update the patch and the process starts again, beginning with
> re-provisioning the builders. But the similarity of most builds to
> others that have gone before seem like a profitable candidate for
> fetching unaffected build results from a cache.
>
> Any thoughts on how one could implement that, assuming I've not
> misunderstood what already exists?  "Repository" is something different,
> right? It seems more oriented to sources, but the doc says it covers
> derived files as well.  And looks like it also takes a "directory",
> rather than a more universal "URL".
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20181029/38d1af3b/attachment.html>