[Scons-users] Thin archive Tool

[Andrew C. Morrow]

Hi -

The SCons based build system for MongoDB makes heavy use of static linking.
One consequence of static linking is that the space requirements are
basically doubled, since each translation unit produces an object file, and
then each object file is copied into an archive file. Adding CacheDir into
the mix multiplies this duplication.

The GNU binutils tools, however, support 'thin' archives, where the archive
contents are simply a list of file references, meaning that the archive
files are very small. At link time, the linker simply dereferences the
listed files in each archive.

To support this, we added the following Apache 2.0 licensed tool:

https://github.com/mongodb/mongo/blob/master/site_scons/site_tools/thin_archive.py

One subtle aspect to consider is that when using thin archives, if
application X depends on libY.a, which contains z.o produced from z.c, then
if z.c is changed, the built-in signature of libY.a will not change, since
the reference to z.o doesn't change when the archive is rebuilt, so taking
the MD5 of the file contents will yield the same result.

To address this, our Tool creates a new Node subclass that overrides the
get_contents and get_content_hash methods, and sets the target_factory for
StaticLibrary to produce that Node subclass. The overriding behavior
computes a content hash based on the content hash of the children of the
new node.

This all seems to work, fairly well, but I was curious if there was a more
appropriate way to accomplish this. The end goal is that we want the
content signature of these Nodes to be a hash of the signatures of all of
the Nodes children, rather than the on-disk contents of the Node.

Is there a better way to accomplish this than what we are doing here?

FWIW we also use a similar technique for driving ABI change linking when
doing dynamic builds:

https://github.com/mongodb/mongo/blob/master/site_scons/site_tools/abilink.py

Here, we would be particularly interested in arranging to interpose in such
a way to absolutely minimize the number of times we must invoke abidw, as
that is very expensive. Avoiding needless re-invocations of abidw as things
move in and out of the CacheDir is desired.

Thanks,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20161020/d76ebd7e/attachment.html>