[Scons-users] Eliding content signature calculation during cache extraction

[Andrew C. Morrow]

Currently, if you extract a file from the cache, the content signature for
that file must be computed. But presumably we knew what the content
signature was when we placed the object in the cache in the first place.

It seems like it should be possible to write the content signature to the
cache along with the object itself. Then, when a needed object is found in
the cache, we can also read back its content signature, and avoid the extra
overhead of recomputing the content signature.

I'm imagining a system where if an object is stored to the cache as
$CACHE_DIR/00/00ea7297f00248e292eaca1bba9999c8, then we would also write a
file $CACHE_DIR/00/00ea7297f00248e292eaca1bba9999c8.csig, containing the
content signature of the object we just inserted.

Then when we go to retrieve an object from the cache, once we have computed
the build signature which points us to a specific object, if we find a
.csig file along with the object itself, we just set the contents of that
.csig file as the content signature for the object, and elide the
computation.

I think this would have the potential to significantly speed up extracting
files from the cache.

Once nice feature of this would be that I think it wouldn't require an
upgrade to the existing cache schema, since if you don't find the .csig
file, you just compute the content signature as you always would have.
Similarly, if you have a .csig file, but the version of SCons you are using
is unaware of the feature, it will ignore it, and again compute the
signature.

The only thing that seems somewhat risky to me is that SCons would now be
reading and writing two files rather than one for each object, so perhaps
there are some issues around atomicity and consistency with multiple
consumers and producers using the same cache? I'd think that is probably
tractable though, with some thought and careful coding.

Does this seem like a useful feature? Does the suggested implementation
seem plausible? Any risks or adverse consequences?

Thanks,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170603/d9e7882a/attachment.html>