[Scons-users] Thin archive Tool
Andrew C. Morrow
andrew.c.morrow at gmail.com
Mon Feb 27 12:54:38 EST 2017
Thanks Bill -
One wrinkle. I defined SHLINKCOMSTR as follows:
env['SHLINKCOMSTR'] = "Via tempfile: $SHLINKCOM"
However, in the output, it appears that the $( $) stuff isn't being elided
as expected:
Via tempfile: /opt/mongodbtoolchain/v2/bin/clang++ -o
build/debug/third_party/libshim_pcrecpp.so -Wl,--no-as-needed -pthread
-Wl,-z,now -rdynamic -Wl,--fatal-warnings -fstack-protector-strong
-fuse-ld=gold -Wl,--build-id -Wl,--hash-style=gnu
-Wl,--detect-odr-violations -Wl,-z,noexecstack -Wl,--warn-execstack
-Wl,-z,relro -shared build/debug/third_party/shim_pcrecpp.os $( $)
-Wl,-z,defs -Wl,--whole-archive -Wl,--start-group
build/debug/third_party/pcre-8.39/libpcrecpp.so -Wl,--end-group
-Wl,--no-whole-archive -lm -lrt -ldl
Is there a way to express that I want "command" (a.k.a raw=2) like
expansion for SHLINKCOMSTR here so that they are dropped?
Thanks,
Andrew
On Mon, Feb 27, 2017 at 11:59 AM, Bill Deegan <bill at baddogconsulting.com>
wrote:
> Andrew,
>
> Indeed. One variable for command line, one for output, for exactly the
> reason you specify.
> You may want or/not want the same for both.
>
> -Bill
>
> On Sun, Feb 26, 2017 at 9:56 PM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>>
>> Ah, I figured out the printing, I think.
>>
>> Normally in a CI build I don't want to set any of the COMSTR variables,
>> because I want the real logging of all commands. But if I have TEMPFILE in
>> play, I *do* want to set a [SH]LINKCOMSTR, and I can just set it to print
>> out $LINKCOM, which gets me the "real" command line. Neat.
>>
>>
>>
>> On Sun, Feb 26, 2017 at 11:22 PM, Andrew C. Morrow <
>> andrew.c.morrow at gmail.com> wrote:
>>
>>>
>>> Hi Bill -
>>>
>>> OK, I did try it, and.... it works! But not right out of the box.
>>>
>>> As it turns out, even though the generic Environment setup in
>>> Platform/__init__.py and the POSIX Platform/posix.py does try to set this
>>> up, the 'link.py' tool (and probably others) don't insert the TEMPFILE bit.
>>> So, it only works on windows, because mslink.py does do it.
>>>
>>> I was able to work around this in my top level SConstruct file by adding
>>> the following after all Tools were loaded:
>>>
>>> if not 'TEMPFILE' in env['LINKCOM']:
>>> env["LINKCOM"] = "${{TEMPFILE('{}', 'LINKCOMSTR')}}".format(env['L
>>> INKCOM'])
>>> if not 'TEMPFILE' in env['SHLINKCOM']:
>>> env["SHLINKCOM"] = "${{TEMPFILE('{}', 'SHLINKCOMSTR')}}".format(env[
>>> 'SHLINKCOM'])
>>>
>>> I think this is probably a bug in SCons?
>>>
>>> One observation about this that still makes me somewhat prefer my
>>> thin_archive approach. With the TEMPFILE approach, the actual build
>>> invocation isn't written to the output, just a message like:
>>>
>>> g++ @/tmp/tmpxGAiil.lnk
>>>
>>> That isn't too great in a CI system, because now the actual build
>>> command we ran isn't printed. Perhaps there is a way to get TempFileMunge
>>> to echo the full command? As you mentioned, it isn't terribly well
>>> documented.
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
>>> On Mon, Oct 24, 2016 at 11:36 PM, Bill Deegan <bill at baddogconsulting.com
>>> > wrote:
>>>
>>>> Did you try using TEMPFILE? and MAXLINELENGTH?
>>>> Which I'm just seeing is far from adequately documented.
>>>>
>>>> see src/engine/SCons/Platform/__init__.py
>>>>
>>>> """A callable class. You can set an Environment variable to this,
>>>> then call it with a string argument, then it will perform temporary
>>>> file substitution on it. This is used to circumvent the long command
>>>> line limitation.
>>>>
>>>> Example usage:
>>>> env["TEMPFILE"] = TempFileMunge
>>>> env["LINKCOM"] = "${TEMPFILE('$LINK $TARGET $SOURCES','$LINKCOMSTR')}"
>>>>
>>>> By default, the name of the temporary file used begins with a
>>>> prefix of '@'. This may be configred for other tool chains by
>>>> setting '$TEMPFILEPREFIX'.
>>>>
>>>> env["TEMPFILEPREFIX"] = '-@' # diab compiler
>>>> env["TEMPFILEPREFIX"] = '-via' # arm tool chain
>>>> """
>>>>
>>>>
>>>>
>>>> On Mon, Oct 24, 2016 at 10:29 PM, Bill Deegan <
>>>> bill at baddogconsulting.com> wrote:
>>>>
>>>>> Andrew,
>>>>>
>>>>> I guess the question is would the extra work be worth it in terms of
>>>>> reduced build time?
>>>>> Would covering all binutils platforms resolve the issue for you and
>>>>> your team?
>>>>> Or would you still be left with needing to use thin archives on some
>>>>> platforms?
>>>>> Which other linkers are you dealing with?
>>>>>
>>>>> -Bill
>>>>>
>>>>> On Mon, Oct 24, 2016 at 4:47 PM, Andrew C. Morrow <
>>>>> andrew.c.morrow at gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> That would probably work on binutils platforms, but I think it would
>>>>>> require generating the linker script from the source list, and then feeding
>>>>>> that into the link step. Without having really investigated, that
>>>>>> intuitively felt like it would be more work to get SCons to do that than
>>>>>> would adding some letters to ARFLAGS and synthesizing a signature for the
>>>>>> target.
>>>>>>
>>>>>> On Mon, Oct 24, 2016 at 2:30 PM, Bill Deegan <
>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>
>>>>>>> Would something like this work?
>>>>>>> https://cygwin.com/ml/cygwin/2004-04/msg00330.html
>>>>>>>
>>>>>>> Or is this for a non-gcc compiler?
>>>>>>> -Bill
>>>>>>>
>>>>>>> On Mon, Oct 24, 2016 at 10:12 AM, Andrew C. Morrow <
>>>>>>> andrew.c.morrow at gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> The short answer is that when linking by enumerating all object
>>>>>>>> files on the command line, we recently hit the hard coded command line
>>>>>>>> length limit on certain linux variants. By using archives to bundle groups
>>>>>>>> of objects into logical libraries, we can sidestep that limitation. The
>>>>>>>> downside was increased IO and disk utilization, but thin archives mitigate
>>>>>>>> that.
>>>>>>>>
>>>>>>>> I'm still curious though - any thoughts on the implementation? Is
>>>>>>>> the mechanism used for interposing on the signature calculations correct?
>>>>>>>> Is there a better, more scons-y way?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 21, 2016 at 8:05 PM, Bill Deegan <
>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>
>>>>>>>>> I'm guessing that using archives let's him skip keeping track of
>>>>>>>>> which object files need to be pulled in (as opposed to listing all the
>>>>>>>>> object files).
>>>>>>>>>
>>>>>>>>> -Bill
>>>>>>>>>
>>>>>>>>> On Fri, Oct 21, 2016 at 11:26 AM, Rob Boehne <robb at datalogics.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Andrew,
>>>>>>>>>>
>>>>>>>>>> Just out of curiosity, why are you using archives at all? There
>>>>>>>>>> are legitimate reasons, and I haven’t looked at MongoDB, but perhaps
>>>>>>>>>> removing this step would simplify your build and solve this issue.
>>>>>>>>>>
>>>>>>>>>> HTH,
>>>>>>>>>>
>>>>>>>>>> Robert Boehne
>>>>>>>>>>
>>>>>>>>>> From: Scons-users <scons-users-bounces at scons.org> on behalf of
>>>>>>>>>> "Andrew C. Morrow" <andrew.c.morrow at gmail.com>
>>>>>>>>>> Reply-To: SCons users mailing list <scons-users at scons.org>
>>>>>>>>>> Date: Thursday, October 20, 2016 at 3:32 PM
>>>>>>>>>> To: SCons users mailing list <scons-users at scons.org>
>>>>>>>>>> Subject: [Scons-users] Thin archive Tool
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi -
>>>>>>>>>>
>>>>>>>>>> The SCons based build system for MongoDB makes heavy use of
>>>>>>>>>> static linking. One consequence of static linking is that the space
>>>>>>>>>> requirements are basically doubled, since each translation unit produces an
>>>>>>>>>> object file, and then each object file is copied into an archive file.
>>>>>>>>>> Adding CacheDir into the mix multiplies this duplication.
>>>>>>>>>>
>>>>>>>>>> The GNU binutils tools, however, support 'thin' archives, where
>>>>>>>>>> the archive contents are simply a list of file references, meaning that the
>>>>>>>>>> archive files are very small. At link time, the linker simply dereferences
>>>>>>>>>> the listed files in each archive.
>>>>>>>>>>
>>>>>>>>>> To support this, we added the following Apache 2.0 licensed tool:
>>>>>>>>>>
>>>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>>>> _tools/thin_archive.py
>>>>>>>>>>
>>>>>>>>>> One subtle aspect to consider is that when using thin archives,
>>>>>>>>>> if application X depends on libY.a, which contains z.o produced from z.c,
>>>>>>>>>> then if z.c is changed, the built-in signature of libY.a will not change,
>>>>>>>>>> since the reference to z.o doesn't change when the archive is rebuilt, so
>>>>>>>>>> taking the MD5 of the file contents will yield the same result.
>>>>>>>>>>
>>>>>>>>>> To address this, our Tool creates a new Node subclass that
>>>>>>>>>> overrides the get_contents and get_content_hash methods, and sets the
>>>>>>>>>> target_factory for StaticLibrary to produce that Node subclass. The
>>>>>>>>>> overriding behavior computes a content hash based on the content hash of
>>>>>>>>>> the children of the new node.
>>>>>>>>>>
>>>>>>>>>> This all seems to work, fairly well, but I was curious if there
>>>>>>>>>> was a more appropriate way to accomplish this. The end goal is that we want
>>>>>>>>>> the content signature of these Nodes to be a hash of the signatures of all
>>>>>>>>>> of the Nodes children, rather than the on-disk contents of the Node.
>>>>>>>>>>
>>>>>>>>>> Is there a better way to accomplish this than what we are doing
>>>>>>>>>> here?
>>>>>>>>>>
>>>>>>>>>> FWIW we also use a similar technique for driving ABI change
>>>>>>>>>> linking when doing dynamic builds:
>>>>>>>>>>
>>>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>>>> _tools/abilink.py
>>>>>>>>>>
>>>>>>>>>> Here, we would be particularly interested in arranging to
>>>>>>>>>> interpose in such a way to absolutely minimize the number of times we must
>>>>>>>>>> invoke abidw, as that is very expensive. Avoiding needless re-invocations
>>>>>>>>>> of abidw as things move in and out of the CacheDir is desired.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Andrew
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Scons-users mailing list
>>>>>>>>>> Scons-users at scons.org
>>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-users mailing list
>>>>>>>>> Scons-users at scons.org
>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170227/0cce3eba/attachment-0001.html>
More information about the Scons-users
mailing list