[Scons-users] Thin archive Tool

[Andrew C. Morrow]

The short answer is that when linking by enumerating all object files on
the command line, we recently hit the hard coded command line length limit
on certain linux variants. By using archives to bundle groups of objects
into logical libraries, we can sidestep that limitation. The downside was
increased IO and disk utilization, but thin archives mitigate that.

I'm still curious though - any thoughts on the implementation? Is the
mechanism used for interposing on the signature calculations correct? Is
there a better, more scons-y way?

Thanks,
Andrew


On Fri, Oct 21, 2016 at 8:05 PM, Bill Deegan <bill at baddogconsulting.com>
wrote:

> I'm guessing that using archives let's him skip keeping track of which
> object files need to be pulled in (as opposed to listing all the object
> files).
>
> -Bill
>
> On Fri, Oct 21, 2016 at 11:26 AM, Rob Boehne <robb at datalogics.com> wrote:
>
>> Andrew,
>>
>> Just out of curiosity, why are you using archives at all?  There are
>> legitimate reasons, and I haven’t looked at MongoDB, but perhaps removing
>> this step would simplify your build and solve this issue.
>>
>> HTH,
>>
>> Robert Boehne
>>
>> From: Scons-users <scons-users-bounces at scons.org> on behalf of "Andrew
>> C. Morrow" <andrew.c.morrow at gmail.com>
>> Reply-To: SCons users mailing list <scons-users at scons.org>
>> Date: Thursday, October 20, 2016 at 3:32 PM
>> To: SCons users mailing list <scons-users at scons.org>
>> Subject: [Scons-users] Thin archive Tool
>>
>>
>> Hi -
>>
>> The SCons based build system for MongoDB makes heavy use of static
>> linking. One consequence of static linking is that the space requirements
>> are basically doubled, since each translation unit produces an object file,
>> and then each object file is copied into an archive file. Adding CacheDir
>> into the mix multiplies this duplication.
>>
>> The GNU binutils tools, however, support 'thin' archives, where the
>> archive contents are simply a list of file references, meaning that the
>> archive files are very small. At link time, the linker simply dereferences
>> the listed files in each archive.
>>
>> To support this, we added the following Apache 2.0 licensed tool:
>>
>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>> _tools/thin_archive.py
>>
>> One subtle aspect to consider is that when using thin archives, if
>> application X depends on libY.a, which contains z.o produced from z.c, then
>> if z.c is changed, the built-in signature of libY.a will not change, since
>> the reference to z.o doesn't change when the archive is rebuilt, so taking
>> the MD5 of the file contents will yield the same result.
>>
>> To address this, our Tool creates a new Node subclass that overrides the
>> get_contents and get_content_hash methods, and sets the target_factory for
>> StaticLibrary to produce that Node subclass. The overriding behavior
>> computes a content hash based on the content hash of the children of the
>> new node.
>>
>> This all seems to work, fairly well, but I was curious if there was a
>> more appropriate way to accomplish this. The end goal is that we want the
>> content signature of these Nodes to be a hash of the signatures of all of
>> the Nodes children, rather than the on-disk contents of the Node.
>>
>> Is there a better way to accomplish this than what we are doing here?
>>
>> FWIW we also use a similar technique for driving ABI change linking when
>> doing dynamic builds:
>>
>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>> _tools/abilink.py
>>
>> Here, we would be particularly interested in arranging to interpose in
>> such a way to absolutely minimize the number of times we must invoke abidw,
>> as that is very expensive. Avoiding needless re-invocations of abidw as
>> things move in and out of the CacheDir is desired.
>>
>> Thanks,
>> Andrew
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20161024/f5e33f63/attachment.html>