[Scons-users] Thin archive Tool

Andrew C. Morrow andrew.c.morrow at gmail.com
Mon Feb 27 00:56:08 EST 2017 

Ah, I figured out the printing, I think.

Normally in a CI build I don't want to set any of the COMSTR variables,
because I want the real logging of all commands. But if I have TEMPFILE in
play, I *do* want to set a [SH]LINKCOMSTR, and I can just set it to print
out $LINKCOM, which gets me the "real" command line. Neat.



On Sun, Feb 26, 2017 at 11:22 PM, Andrew C. Morrow <
andrew.c.morrow at gmail.com> wrote:

>
> Hi Bill -
>
> OK, I did try it, and.... it works! But not right out of the box.
>
> As it turns out, even though the generic Environment setup in
> Platform/__init__.py and the POSIX Platform/posix.py does try to set this
> up, the 'link.py' tool (and probably others) don't insert the TEMPFILE bit.
> So, it only works on windows, because mslink.py does do it.
>
> I was able to work around this in my top level SConstruct file by adding
> the following after all Tools were loaded:
>
> if not 'TEMPFILE' in env['LINKCOM']:
>     env["LINKCOM"] = "${{TEMPFILE('{}', 'LINKCOMSTR')}}".format(env['
> LINKCOM'])
> if not 'TEMPFILE' in env['SHLINKCOM']:
>     env["SHLINKCOM"] = "${{TEMPFILE('{}', 'SHLINKCOMSTR')}}".format(env[
> 'SHLINKCOM'])
>
> I think this is probably a bug in SCons?
>
> One observation about this that still makes me somewhat prefer my
> thin_archive approach. With the TEMPFILE approach, the actual build
> invocation isn't written to the output, just a message like:
>
> g++ @/tmp/tmpxGAiil.lnk
>
> That isn't too great in a CI system, because now the actual build command
> we ran isn't printed. Perhaps there is a way to get TempFileMunge to echo
> the full command? As you mentioned, it isn't terribly well documented.
>
> Thanks,
> Andrew
>
>
> On Mon, Oct 24, 2016 at 11:36 PM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> Did you try using TEMPFILE? and MAXLINELENGTH?
>> Which I'm just seeing is far from adequately documented.
>>
>> see src/engine/SCons/Platform/__init__.py
>>
>> """A callable class.  You can set an Environment variable to this,
>> then call it with a string argument, then it will perform temporary
>> file substitution on it.  This is used to circumvent the long command
>> line limitation.
>>
>> Example usage:
>> env["TEMPFILE"] = TempFileMunge
>> env["LINKCOM"] = "${TEMPFILE('$LINK $TARGET $SOURCES','$LINKCOMSTR')}"
>>
>> By default, the name of the temporary file used begins with a
>> prefix of '@'.  This may be configred for other tool chains by
>> setting '$TEMPFILEPREFIX'.
>>
>> env["TEMPFILEPREFIX"] = '-@'        # diab compiler
>> env["TEMPFILEPREFIX"] = '-via'      # arm tool chain
>> """
>>
>>
>>
>> On Mon, Oct 24, 2016 at 10:29 PM, Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>>> Andrew,
>>>
>>> I guess the question is would the extra work be worth it in terms of
>>> reduced build time?
>>> Would covering all binutils platforms resolve the issue for you and your
>>> team?
>>> Or would you still be left with needing to use  thin archives on some
>>> platforms?
>>> Which other linkers are you dealing with?
>>>
>>> -Bill
>>>
>>> On Mon, Oct 24, 2016 at 4:47 PM, Andrew C. Morrow <
>>> andrew.c.morrow at gmail.com> wrote:
>>>
>>>>
>>>> That would probably work on binutils platforms, but I think it would
>>>> require generating the linker script from the source list, and then feeding
>>>> that into the link step. Without having really investigated, that
>>>> intuitively felt like it would be more work to get SCons to do that than
>>>> would adding some letters to ARFLAGS and synthesizing a signature for the
>>>> target.
>>>>
>>>> On Mon, Oct 24, 2016 at 2:30 PM, Bill Deegan <bill at baddogconsulting.com
>>>> > wrote:
>>>>
>>>>> Would something like this work?
>>>>> https://cygwin.com/ml/cygwin/2004-04/msg00330.html
>>>>>
>>>>> Or is this for a non-gcc compiler?
>>>>> -Bill
>>>>>
>>>>> On Mon, Oct 24, 2016 at 10:12 AM, Andrew C. Morrow <
>>>>> andrew.c.morrow at gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> The short answer is that when linking by enumerating all object files
>>>>>> on the command line, we recently hit the hard coded command line length
>>>>>> limit on certain linux variants. By using archives to bundle groups of
>>>>>> objects into logical libraries, we can sidestep that limitation. The
>>>>>> downside was increased IO and disk utilization, but thin archives mitigate
>>>>>> that.
>>>>>>
>>>>>> I'm still curious though - any thoughts on the implementation? Is the
>>>>>> mechanism used for interposing on the signature calculations correct? Is
>>>>>> there a better, more scons-y way?
>>>>>>
>>>>>> Thanks,
>>>>>> Andrew
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 21, 2016 at 8:05 PM, Bill Deegan <
>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>
>>>>>>> I'm guessing that using archives let's him skip keeping track of
>>>>>>> which object files need to be pulled in (as opposed to listing all the
>>>>>>> object files).
>>>>>>>
>>>>>>> -Bill
>>>>>>>
>>>>>>> On Fri, Oct 21, 2016 at 11:26 AM, Rob Boehne <robb at datalogics.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Andrew,
>>>>>>>>
>>>>>>>> Just out of curiosity, why are you using archives at all?  There
>>>>>>>> are legitimate reasons, and I haven’t looked at MongoDB, but perhaps
>>>>>>>> removing this step would simplify your build and solve this issue.
>>>>>>>>
>>>>>>>> HTH,
>>>>>>>>
>>>>>>>> Robert Boehne
>>>>>>>>
>>>>>>>> From: Scons-users <scons-users-bounces at scons.org> on behalf of
>>>>>>>> "Andrew C. Morrow" <andrew.c.morrow at gmail.com>
>>>>>>>> Reply-To: SCons users mailing list <scons-users at scons.org>
>>>>>>>> Date: Thursday, October 20, 2016 at 3:32 PM
>>>>>>>> To: SCons users mailing list <scons-users at scons.org>
>>>>>>>> Subject: [Scons-users] Thin archive Tool
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi -
>>>>>>>>
>>>>>>>> The SCons based build system for MongoDB makes heavy use of static
>>>>>>>> linking. One consequence of static linking is that the space requirements
>>>>>>>> are basically doubled, since each translation unit produces an object file,
>>>>>>>> and then each object file is copied into an archive file. Adding CacheDir
>>>>>>>> into the mix multiplies this duplication.
>>>>>>>>
>>>>>>>> The GNU binutils tools, however, support 'thin' archives, where the
>>>>>>>> archive contents are simply a list of file references, meaning that the
>>>>>>>> archive files are very small. At link time, the linker simply dereferences
>>>>>>>> the listed files in each archive.
>>>>>>>>
>>>>>>>> To support this, we added the following Apache 2.0 licensed tool:
>>>>>>>>
>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>> _tools/thin_archive.py
>>>>>>>>
>>>>>>>> One subtle aspect to consider is that when using thin archives, if
>>>>>>>> application X depends on libY.a, which contains z.o produced from z.c, then
>>>>>>>> if z.c is changed, the built-in signature of libY.a will not change, since
>>>>>>>> the reference to z.o doesn't change when the archive is rebuilt, so taking
>>>>>>>> the MD5 of the file contents will yield the same result.
>>>>>>>>
>>>>>>>> To address this, our Tool creates a new Node subclass that
>>>>>>>> overrides the get_contents and get_content_hash methods, and sets the
>>>>>>>> target_factory for StaticLibrary to produce that Node subclass. The
>>>>>>>> overriding behavior computes a content hash based on the content hash of
>>>>>>>> the children of the new node.
>>>>>>>>
>>>>>>>> This all seems to work, fairly well, but I was curious if there was
>>>>>>>> a more appropriate way to accomplish this. The end goal is that we want the
>>>>>>>> content signature of these Nodes to be a hash of the signatures of all of
>>>>>>>> the Nodes children, rather than the on-disk contents of the Node.
>>>>>>>>
>>>>>>>> Is there a better way to accomplish this than what we are doing
>>>>>>>> here?
>>>>>>>>
>>>>>>>> FWIW we also use a similar technique for driving ABI change linking
>>>>>>>> when doing dynamic builds:
>>>>>>>>
>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>> _tools/abilink.py
>>>>>>>>
>>>>>>>> Here, we would be particularly interested in arranging to interpose
>>>>>>>> in such a way to absolutely minimize the number of times we must invoke
>>>>>>>> abidw, as that is very expensive. Avoiding needless re-invocations of abidw
>>>>>>>> as things move in and out of the CacheDir is desired.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170227/0d6798cb/attachment-0001.html>

More information about the Scons-users mailing list