[Scons-users] scons cache in a CI scenario

[Bill Deegan]

Copying from cache vs building is hard to answer in general as you say.
Depends on speed of filesystem( SSD or spinning disk or network), and also
speed of compiles.

In theory you should never get a bad build where something doesn't rebuild
which doesn't.
At this point I'm not aware of any outstanding bugs which lead to this.
There are outstanding bugs where something rebuilds when not necessary.

SCons's philosophy is to never allow incorrect builds.  Secondarily, avoid
rebuilding when not necessary.

In general using cache is a win for most builds, especially where you have
a CI build doing nightlies your developer builds can pull from when
rebuilding. (Common use model).

Also, most don't use cached builds for production, but mainly for developer
builds.

Yes you can disable and enable using the cache via command line flags.

-Bill


On Wed, Oct 31, 2018 at 8:35 AM Mats Wichmann <mats at wichmann.us> wrote:

> On 10/29/18 11:19 AM, Andrew C. Morrow wrote:
> > My company has done this by using Amazon EFS <
> https://aws.amazon.com/efs/>
> > to back the SCons cache for the parts of our CI system that run in AWS,
> and
> > the results have been excellent.
> >
> > We create a new cache directory on EFS for each new build image version
> > that we run, so that changes to the build image that may not be visible
> to
> > SCons are not a concern. We discussed at one point making it so that the
> > SCons cache could be backed by something other than a filesystem, but EFS
> > served well enough for us that we never pursued that further. It is also
> > important to consider how you will prune the cache, or identify old
> caches
> > that are no longer in active use, so that the EFS footprint doesn't grow
> > without bound. We wrote scripts for that which evaluate the most recent
> > access time of the files (and we had to make small modification to SCons
> to
> > get the timestamps right) and only retain the most recently accessed
> files
> > in the cache.
> >
> > Overall, It is definitely a project worth pursuing to reduce CI build
> > times: it isn't trivial to set up, but the build time savings were for us
> > very real.
> >
> > Thanks,
> > Andrew
>
> I didn't respond and meant to: thanks for this info!  I believe in our
> case (I checked with the host of our infra) it's possible to set
> something up in OpenStack that will appear as part of the filesystem of
> the instances.
>
> I got some surprising pushback when bringing up this possibility in the
> project - I brought it up because frankly I'm tired of spending two
> hours or more waiting for an answer from the CI system... something that
> isn't really important to the end product, but is important to the
> developer experience. I'll just paste the relevant bits instead of
> paraphrasing:
>
> =========
> a cached build is not a good idea.
>
> Number one reason for not using a cached build. "Reliability".
>   - Case 1: A problem is solved just by doing a clean build.
>   - Case 2: A problem is hidden because a left over build product from
> the cached build items.
>
> A false failure like Case 1 where the build fails but can be fixed with
> a clean build is annoying but not a critical problem as long as there is
> a way to force a clean build. The big problem is Case 2 where it passes
> because of picking up something from the cached build. The error will,
> likely, not be caught till it is merged. Both problems I agree are
> unusual but they still happen.
>
> I am all for speeding up the build but I suspect adding cached build
> would cause more problems than its worth.
> =========
>
> I still don't fully grok how the various signatures are generated, which
> I presume is the answer to the reliability question.  I presume it would
> be really hard to get a "wrong answer" in the scons scenario?
>
> I have also worried about what Andrew mentioned - not as a "stopper" but
> just something that needs doing - how do you keep the cache fresh and
> not just growing over time with eventually unneeded artifacts, etc.
>
> Other (cough) builsystems seem to have paid a lot of attention to
> sharing build artifacts through caching.  In particular:
>
> https://docs.bazel.build/versions/master/remote-caching.html
>
> Scons does not seem to have any concept of a networked cache - it would
> work if you had the network connected to the local filesystem or the
> schemes already mentioned in this thread, but to the viewpoint of SCons
> that's all local.  Does thinking about a network cache make any sense?
> I'd worry that time-deciding-if-we're-going-to-use followed by
> time-actually-fetching may often be more than just building locally,
> esp. if you follow a programming model that leaves you with relatively
> small source files.  I'm happy to stop thinking about it if we know it's
> not a particularly profitable model :)
>
> "Problem solved by doing a clean build" - I presume that would be giving
> the --cache-disable flag, since otherwise the files "come back" after a
> clean, right?
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20181031/eb6868ea/attachment.html>