[Scons-users] Thin archive Tool

[Bill Deegan]

Would something like this work?
https://cygwin.com/ml/cygwin/2004-04/msg00330.html

Or is this for a non-gcc compiler?
-Bill

On Mon, Oct 24, 2016 at 10:12 AM, Andrew C. Morrow <
andrew.c.morrow at gmail.com> wrote:

>
> The short answer is that when linking by enumerating all object files on
> the command line, we recently hit the hard coded command line length limit
> on certain linux variants. By using archives to bundle groups of objects
> into logical libraries, we can sidestep that limitation. The downside was
> increased IO and disk utilization, but thin archives mitigate that.
>
> I'm still curious though - any thoughts on the implementation? Is the
> mechanism used for interposing on the signature calculations correct? Is
> there a better, more scons-y way?
>
> Thanks,
> Andrew
>
>
> On Fri, Oct 21, 2016 at 8:05 PM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> I'm guessing that using archives let's him skip keeping track of which
>> object files need to be pulled in (as opposed to listing all the object
>> files).
>>
>> -Bill
>>
>> On Fri, Oct 21, 2016 at 11:26 AM, Rob Boehne <robb at datalogics.com> wrote:
>>
>>> Andrew,
>>>
>>> Just out of curiosity, why are you using archives at all?  There are
>>> legitimate reasons, and I haven’t looked at MongoDB, but perhaps removing
>>> this step would simplify your build and solve this issue.
>>>
>>> HTH,
>>>
>>> Robert Boehne
>>>
>>> From: Scons-users <scons-users-bounces at scons.org> on behalf of "Andrew
>>> C. Morrow" <andrew.c.morrow at gmail.com>
>>> Reply-To: SCons users mailing list <scons-users at scons.org>
>>> Date: Thursday, October 20, 2016 at 3:32 PM
>>> To: SCons users mailing list <scons-users at scons.org>
>>> Subject: [Scons-users] Thin archive Tool
>>>
>>>
>>> Hi -
>>>
>>> The SCons based build system for MongoDB makes heavy use of static
>>> linking. One consequence of static linking is that the space requirements
>>> are basically doubled, since each translation unit produces an object file,
>>> and then each object file is copied into an archive file. Adding CacheDir
>>> into the mix multiplies this duplication.
>>>
>>> The GNU binutils tools, however, support 'thin' archives, where the
>>> archive contents are simply a list of file references, meaning that the
>>> archive files are very small. At link time, the linker simply dereferences
>>> the listed files in each archive.
>>>
>>> To support this, we added the following Apache 2.0 licensed tool:
>>>
>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>> _tools/thin_archive.py
>>>
>>> One subtle aspect to consider is that when using thin archives, if
>>> application X depends on libY.a, which contains z.o produced from z.c, then
>>> if z.c is changed, the built-in signature of libY.a will not change, since
>>> the reference to z.o doesn't change when the archive is rebuilt, so taking
>>> the MD5 of the file contents will yield the same result.
>>>
>>> To address this, our Tool creates a new Node subclass that overrides the
>>> get_contents and get_content_hash methods, and sets the target_factory for
>>> StaticLibrary to produce that Node subclass. The overriding behavior
>>> computes a content hash based on the content hash of the children of the
>>> new node.
>>>
>>> This all seems to work, fairly well, but I was curious if there was a
>>> more appropriate way to accomplish this. The end goal is that we want the
>>> content signature of these Nodes to be a hash of the signatures of all of
>>> the Nodes children, rather than the on-disk contents of the Node.
>>>
>>> Is there a better way to accomplish this than what we are doing here?
>>>
>>> FWIW we also use a similar technique for driving ABI change linking when
>>> doing dynamic builds:
>>>
>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>> _tools/abilink.py
>>>
>>> Here, we would be particularly interested in arranging to interpose in
>>> such a way to absolutely minimize the number of times we must invoke abidw,
>>> as that is very expensive. Avoiding needless re-invocations of abidw as
>>> things move in and out of the CacheDir is desired.
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20161024/45a4071a/attachment-0001.html>