[Scons-users] Adding support for policy based CacheDir pruning?

[Andrew C. Morrow]

I recently started using the 'CacheDir' feature of SCons, and it is nothing
short of miraculous in terms of build time savings for C++ projects.
Checking out a topic branch which alters a widely included header then
switching back to master is no longer something to dread.

However, for a large project which generates hundreds of .o files and some
100+ statically linked targets each pulling in a large subset of those .o
files, the cache can grow very large, even when only building a few
different branches or when building in a few different modes (debug/ndebug,
opt/nopt). Multiplying branches by build variants causes even faster cache
growth. I've routinely seen cache directories grow well beyond 100GB after
only a few builds.

I also have some anecdotal evidence that once the cache has grown beyond a
certain size it begins to slow down the build rather than speed it up. I
haven't done any timing yet, but others on my team experimenting with
CacheDir share this anecdotal experience.

I am aware of this suggestion:
http://scons.org/wiki/LimitCacheSizeWithProgress

However, there are some problems with that approach:

- It abuses the Progress mechanism.
- It is very much a python solution to the problem, rather than a SCons
solution.
- It conflates the idea of identifying stale cache items with their
physical removal.

I think the utility of CacheDir would be greatly improved by implementing
some sort of built-in cache eviction mechanism. I think a good solution to
this would:

- Allow the injection of a user designed cache limit policy.
- Provide a default policy that capped the cache size at some modest limit
and evicted in LRU or age order, with the size cap and age limit easily
specified by the user.
- Handle cache entry removal within SCons, based on the results of
evaluating the currently active policy. The policy should not need to
interact with the actual files in the cache or know anything about the
cache internals.

Is there interest in such a feature? Has it been attempted before? Would
the SCons maintainers be willing to consider patches to add this sort of
feature? Roadmap? I'd be willing to do some work towards this, but only if
it seems possible, and stands some reasonable chance of becoming part of
the official SCons project.

Thanks,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://four.pairlist.net/pipermail/scons-users/attachments/20131017/b0dc62cf/attachment.htm