[Scons-users] Thin archive Tool
Andrew C. Morrow
andrew.c.morrow at gmail.com
Sun Feb 26 23:22:37 EST 2017
Hi Bill -
OK, I did try it, and.... it works! But not right out of the box.
As it turns out, even though the generic Environment setup in
Platform/__init__.py and the POSIX Platform/posix.py does try to set this
up, the 'link.py' tool (and probably others) don't insert the TEMPFILE bit.
So, it only works on windows, because mslink.py does do it.
I was able to work around this in my top level SConstruct file by adding
the following after all Tools were loaded:
if not 'TEMPFILE' in env['LINKCOM']:
env["LINKCOM"] = "${{TEMPFILE('{}',
'LINKCOMSTR')}}".format(env['LINKCOM'])
if not 'TEMPFILE' in env['SHLINKCOM']:
env["SHLINKCOM"] = "${{TEMPFILE('{}',
'SHLINKCOMSTR')}}".format(env['SHLINKCOM'])
I think this is probably a bug in SCons?
One observation about this that still makes me somewhat prefer my
thin_archive approach. With the TEMPFILE approach, the actual build
invocation isn't written to the output, just a message like:
g++ @/tmp/tmpxGAiil.lnk
That isn't too great in a CI system, because now the actual build command
we ran isn't printed. Perhaps there is a way to get TempFileMunge to echo
the full command? As you mentioned, it isn't terribly well documented.
Thanks,
Andrew
On Mon, Oct 24, 2016 at 11:36 PM, Bill Deegan <bill at baddogconsulting.com>
wrote:
> Did you try using TEMPFILE? and MAXLINELENGTH?
> Which I'm just seeing is far from adequately documented.
>
> see src/engine/SCons/Platform/__init__.py
>
> """A callable class. You can set an Environment variable to this,
> then call it with a string argument, then it will perform temporary
> file substitution on it. This is used to circumvent the long command
> line limitation.
>
> Example usage:
> env["TEMPFILE"] = TempFileMunge
> env["LINKCOM"] = "${TEMPFILE('$LINK $TARGET $SOURCES','$LINKCOMSTR')}"
>
> By default, the name of the temporary file used begins with a
> prefix of '@'. This may be configred for other tool chains by
> setting '$TEMPFILEPREFIX'.
>
> env["TEMPFILEPREFIX"] = '-@' # diab compiler
> env["TEMPFILEPREFIX"] = '-via' # arm tool chain
> """
>
>
>
> On Mon, Oct 24, 2016 at 10:29 PM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> Andrew,
>>
>> I guess the question is would the extra work be worth it in terms of
>> reduced build time?
>> Would covering all binutils platforms resolve the issue for you and your
>> team?
>> Or would you still be left with needing to use thin archives on some
>> platforms?
>> Which other linkers are you dealing with?
>>
>> -Bill
>>
>> On Mon, Oct 24, 2016 at 4:47 PM, Andrew C. Morrow <
>> andrew.c.morrow at gmail.com> wrote:
>>
>>>
>>> That would probably work on binutils platforms, but I think it would
>>> require generating the linker script from the source list, and then feeding
>>> that into the link step. Without having really investigated, that
>>> intuitively felt like it would be more work to get SCons to do that than
>>> would adding some letters to ARFLAGS and synthesizing a signature for the
>>> target.
>>>
>>> On Mon, Oct 24, 2016 at 2:30 PM, Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>>> Would something like this work?
>>>> https://cygwin.com/ml/cygwin/2004-04/msg00330.html
>>>>
>>>> Or is this for a non-gcc compiler?
>>>> -Bill
>>>>
>>>> On Mon, Oct 24, 2016 at 10:12 AM, Andrew C. Morrow <
>>>> andrew.c.morrow at gmail.com> wrote:
>>>>
>>>>>
>>>>> The short answer is that when linking by enumerating all object files
>>>>> on the command line, we recently hit the hard coded command line length
>>>>> limit on certain linux variants. By using archives to bundle groups of
>>>>> objects into logical libraries, we can sidestep that limitation. The
>>>>> downside was increased IO and disk utilization, but thin archives mitigate
>>>>> that.
>>>>>
>>>>> I'm still curious though - any thoughts on the implementation? Is the
>>>>> mechanism used for interposing on the signature calculations correct? Is
>>>>> there a better, more scons-y way?
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>>
>>>>> On Fri, Oct 21, 2016 at 8:05 PM, Bill Deegan <
>>>>> bill at baddogconsulting.com> wrote:
>>>>>
>>>>>> I'm guessing that using archives let's him skip keeping track of
>>>>>> which object files need to be pulled in (as opposed to listing all the
>>>>>> object files).
>>>>>>
>>>>>> -Bill
>>>>>>
>>>>>> On Fri, Oct 21, 2016 at 11:26 AM, Rob Boehne <robb at datalogics.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Andrew,
>>>>>>>
>>>>>>> Just out of curiosity, why are you using archives at all? There are
>>>>>>> legitimate reasons, and I haven’t looked at MongoDB, but perhaps removing
>>>>>>> this step would simplify your build and solve this issue.
>>>>>>>
>>>>>>> HTH,
>>>>>>>
>>>>>>> Robert Boehne
>>>>>>>
>>>>>>> From: Scons-users <scons-users-bounces at scons.org> on behalf of
>>>>>>> "Andrew C. Morrow" <andrew.c.morrow at gmail.com>
>>>>>>> Reply-To: SCons users mailing list <scons-users at scons.org>
>>>>>>> Date: Thursday, October 20, 2016 at 3:32 PM
>>>>>>> To: SCons users mailing list <scons-users at scons.org>
>>>>>>> Subject: [Scons-users] Thin archive Tool
>>>>>>>
>>>>>>>
>>>>>>> Hi -
>>>>>>>
>>>>>>> The SCons based build system for MongoDB makes heavy use of static
>>>>>>> linking. One consequence of static linking is that the space requirements
>>>>>>> are basically doubled, since each translation unit produces an object file,
>>>>>>> and then each object file is copied into an archive file. Adding CacheDir
>>>>>>> into the mix multiplies this duplication.
>>>>>>>
>>>>>>> The GNU binutils tools, however, support 'thin' archives, where the
>>>>>>> archive contents are simply a list of file references, meaning that the
>>>>>>> archive files are very small. At link time, the linker simply dereferences
>>>>>>> the listed files in each archive.
>>>>>>>
>>>>>>> To support this, we added the following Apache 2.0 licensed tool:
>>>>>>>
>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>> _tools/thin_archive.py
>>>>>>>
>>>>>>> One subtle aspect to consider is that when using thin archives, if
>>>>>>> application X depends on libY.a, which contains z.o produced from z.c, then
>>>>>>> if z.c is changed, the built-in signature of libY.a will not change, since
>>>>>>> the reference to z.o doesn't change when the archive is rebuilt, so taking
>>>>>>> the MD5 of the file contents will yield the same result.
>>>>>>>
>>>>>>> To address this, our Tool creates a new Node subclass that overrides
>>>>>>> the get_contents and get_content_hash methods, and sets the target_factory
>>>>>>> for StaticLibrary to produce that Node subclass. The overriding behavior
>>>>>>> computes a content hash based on the content hash of the children of the
>>>>>>> new node.
>>>>>>>
>>>>>>> This all seems to work, fairly well, but I was curious if there was
>>>>>>> a more appropriate way to accomplish this. The end goal is that we want the
>>>>>>> content signature of these Nodes to be a hash of the signatures of all of
>>>>>>> the Nodes children, rather than the on-disk contents of the Node.
>>>>>>>
>>>>>>> Is there a better way to accomplish this than what we are doing here?
>>>>>>>
>>>>>>> FWIW we also use a similar technique for driving ABI change linking
>>>>>>> when doing dynamic builds:
>>>>>>>
>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>> _tools/abilink.py
>>>>>>>
>>>>>>> Here, we would be particularly interested in arranging to interpose
>>>>>>> in such a way to absolutely minimize the number of times we must invoke
>>>>>>> abidw, as that is very expensive. Avoiding needless re-invocations of abidw
>>>>>>> as things move in and out of the CacheDir is desired.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170226/813f45d2/attachment-0001.html>
More information about the Scons-users
mailing list