[Scons-users] Thin archive Tool

Bill Deegan bill at baddogconsulting.com
Mon Feb 27 11:59:54 EST 2017 

Andrew,

Indeed. One variable for command line, one for output, for exactly the
reason you specify.
You may want or/not want the same for both.

-Bill

On Sun, Feb 26, 2017 at 9:56 PM, Andrew C. Morrow <andrew.c.morrow at gmail.com
> wrote:

>
> Ah, I figured out the printing, I think.
>
> Normally in a CI build I don't want to set any of the COMSTR variables,
> because I want the real logging of all commands. But if I have TEMPFILE in
> play, I *do* want to set a [SH]LINKCOMSTR, and I can just set it to print
> out $LINKCOM, which gets me the "real" command line. Neat.
>
>
>
> On Sun, Feb 26, 2017 at 11:22 PM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>>
>> Hi Bill -
>>
>> OK, I did try it, and.... it works! But not right out of the box.
>>
>> As it turns out, even though the generic Environment setup in
>> Platform/__init__.py and the POSIX Platform/posix.py does try to set this
>> up, the 'link.py' tool (and probably others) don't insert the TEMPFILE bit.
>> So, it only works on windows, because mslink.py does do it.
>>
>> I was able to work around this in my top level SConstruct file by adding
>> the following after all Tools were loaded:
>>
>> if not 'TEMPFILE' in env['LINKCOM']:
>>     env["LINKCOM"] = "${{TEMPFILE('{}', 'LINKCOMSTR')}}".format(env['L
>> INKCOM'])
>> if not 'TEMPFILE' in env['SHLINKCOM']:
>>     env["SHLINKCOM"] = "${{TEMPFILE('{}', 'SHLINKCOMSTR')}}".format(env[
>> 'SHLINKCOM'])
>>
>> I think this is probably a bug in SCons?
>>
>> One observation about this that still makes me somewhat prefer my
>> thin_archive approach. With the TEMPFILE approach, the actual build
>> invocation isn't written to the output, just a message like:
>>
>> g++ @/tmp/tmpxGAiil.lnk
>>
>> That isn't too great in a CI system, because now the actual build command
>> we ran isn't printed. Perhaps there is a way to get TempFileMunge to echo
>> the full command? As you mentioned, it isn't terribly well documented.
>>
>> Thanks,
>> Andrew
>>
>>
>> On Mon, Oct 24, 2016 at 11:36 PM, Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>>> Did you try using TEMPFILE? and MAXLINELENGTH?
>>> Which I'm just seeing is far from adequately documented.
>>>
>>> see src/engine/SCons/Platform/__init__.py
>>>
>>> """A callable class.  You can set an Environment variable to this,
>>> then call it with a string argument, then it will perform temporary
>>> file substitution on it.  This is used to circumvent the long command
>>> line limitation.
>>>
>>> Example usage:
>>> env["TEMPFILE"] = TempFileMunge
>>> env["LINKCOM"] = "${TEMPFILE('$LINK $TARGET $SOURCES','$LINKCOMSTR')}"
>>>
>>> By default, the name of the temporary file used begins with a
>>> prefix of '@'.  This may be configred for other tool chains by
>>> setting '$TEMPFILEPREFIX'.
>>>
>>> env["TEMPFILEPREFIX"] = '-@'        # diab compiler
>>> env["TEMPFILEPREFIX"] = '-via'      # arm tool chain
>>> """
>>>
>>>
>>>
>>> On Mon, Oct 24, 2016 at 10:29 PM, Bill Deegan <bill at baddogconsulting.com
>>> > wrote:
>>>
>>>> Andrew,
>>>>
>>>> I guess the question is would the extra work be worth it in terms of
>>>> reduced build time?
>>>> Would covering all binutils platforms resolve the issue for you and
>>>> your team?
>>>> Or would you still be left with needing to use  thin archives on some
>>>> platforms?
>>>> Which other linkers are you dealing with?
>>>>
>>>> -Bill
>>>>
>>>> On Mon, Oct 24, 2016 at 4:47 PM, Andrew C. Morrow <
>>>> andrew.c.morrow at gmail.com> wrote:
>>>>
>>>>>
>>>>> That would probably work on binutils platforms, but I think it would
>>>>> require generating the linker script from the source list, and then feeding
>>>>> that into the link step. Without having really investigated, that
>>>>> intuitively felt like it would be more work to get SCons to do that than
>>>>> would adding some letters to ARFLAGS and synthesizing a signature for the
>>>>> target.
>>>>>
>>>>> On Mon, Oct 24, 2016 at 2:30 PM, Bill Deegan <
>>>>> bill at baddogconsulting.com> wrote:
>>>>>
>>>>>> Would something like this work?
>>>>>> https://cygwin.com/ml/cygwin/2004-04/msg00330.html
>>>>>>
>>>>>> Or is this for a non-gcc compiler?
>>>>>> -Bill
>>>>>>
>>>>>> On Mon, Oct 24, 2016 at 10:12 AM, Andrew C. Morrow <
>>>>>> andrew.c.morrow at gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>> The short answer is that when linking by enumerating all object
>>>>>>> files on the command line, we recently hit the hard coded command line
>>>>>>> length limit on certain linux variants. By using archives to bundle groups
>>>>>>> of objects into logical libraries, we can sidestep that limitation. The
>>>>>>> downside was increased IO and disk utilization, but thin archives mitigate
>>>>>>> that.
>>>>>>>
>>>>>>> I'm still curious though - any thoughts on the implementation? Is
>>>>>>> the mechanism used for interposing on the signature calculations correct?
>>>>>>> Is there a better, more scons-y way?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 21, 2016 at 8:05 PM, Bill Deegan <
>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>
>>>>>>>> I'm guessing that using archives let's him skip keeping track of
>>>>>>>> which object files need to be pulled in (as opposed to listing all the
>>>>>>>> object files).
>>>>>>>>
>>>>>>>> -Bill
>>>>>>>>
>>>>>>>> On Fri, Oct 21, 2016 at 11:26 AM, Rob Boehne <robb at datalogics.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Andrew,
>>>>>>>>>
>>>>>>>>> Just out of curiosity, why are you using archives at all?  There
>>>>>>>>> are legitimate reasons, and I haven’t looked at MongoDB, but perhaps
>>>>>>>>> removing this step would simplify your build and solve this issue.
>>>>>>>>>
>>>>>>>>> HTH,
>>>>>>>>>
>>>>>>>>> Robert Boehne
>>>>>>>>>
>>>>>>>>> From: Scons-users <scons-users-bounces at scons.org> on behalf of
>>>>>>>>> "Andrew C. Morrow" <andrew.c.morrow at gmail.com>
>>>>>>>>> Reply-To: SCons users mailing list <scons-users at scons.org>
>>>>>>>>> Date: Thursday, October 20, 2016 at 3:32 PM
>>>>>>>>> To: SCons users mailing list <scons-users at scons.org>
>>>>>>>>> Subject: [Scons-users] Thin archive Tool
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi -
>>>>>>>>>
>>>>>>>>> The SCons based build system for MongoDB makes heavy use of static
>>>>>>>>> linking. One consequence of static linking is that the space requirements
>>>>>>>>> are basically doubled, since each translation unit produces an object file,
>>>>>>>>> and then each object file is copied into an archive file. Adding CacheDir
>>>>>>>>> into the mix multiplies this duplication.
>>>>>>>>>
>>>>>>>>> The GNU binutils tools, however, support 'thin' archives, where
>>>>>>>>> the archive contents are simply a list of file references, meaning that the
>>>>>>>>> archive files are very small. At link time, the linker simply dereferences
>>>>>>>>> the listed files in each archive.
>>>>>>>>>
>>>>>>>>> To support this, we added the following Apache 2.0 licensed tool:
>>>>>>>>>
>>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>>> _tools/thin_archive.py
>>>>>>>>>
>>>>>>>>> One subtle aspect to consider is that when using thin archives, if
>>>>>>>>> application X depends on libY.a, which contains z.o produced from z.c, then
>>>>>>>>> if z.c is changed, the built-in signature of libY.a will not change, since
>>>>>>>>> the reference to z.o doesn't change when the archive is rebuilt, so taking
>>>>>>>>> the MD5 of the file contents will yield the same result.
>>>>>>>>>
>>>>>>>>> To address this, our Tool creates a new Node subclass that
>>>>>>>>> overrides the get_contents and get_content_hash methods, and sets the
>>>>>>>>> target_factory for StaticLibrary to produce that Node subclass. The
>>>>>>>>> overriding behavior computes a content hash based on the content hash of
>>>>>>>>> the children of the new node.
>>>>>>>>>
>>>>>>>>> This all seems to work, fairly well, but I was curious if there
>>>>>>>>> was a more appropriate way to accomplish this. The end goal is that we want
>>>>>>>>> the content signature of these Nodes to be a hash of the signatures of all
>>>>>>>>> of the Nodes children, rather than the on-disk contents of the Node.
>>>>>>>>>
>>>>>>>>> Is there a better way to accomplish this than what we are doing
>>>>>>>>> here?
>>>>>>>>>
>>>>>>>>> FWIW we also use a similar technique for driving ABI change
>>>>>>>>> linking when doing dynamic builds:
>>>>>>>>>
>>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>>> _tools/abilink.py
>>>>>>>>>
>>>>>>>>> Here, we would be particularly interested in arranging to
>>>>>>>>> interpose in such a way to absolutely minimize the number of times we must
>>>>>>>>> invoke abidw, as that is very expensive. Avoiding needless re-invocations
>>>>>>>>> of abidw as things move in and out of the CacheDir is desired.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-users mailing list
>>>>>>>>> Scons-users at scons.org
>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170227/1d3be78f/attachment-0001.html>

More information about the Scons-users mailing list