[Scons-users] Thin archive Tool

Bill Deegan bill at baddogconsulting.com
Mon Feb 27 13:53:48 EST 2017 

Andrew,

If you use something like this (From manpage). you can specify foo which
removes the $( and $)'s from the argument..
That's likely the simplest way.

Though typically the $('s aren't passed to the *COMSTR's.

You can use this feature to pass arguments to a Python function by creating
a callable class that stores one or more arguments in an object, and then
uses them when the __call__() method is called. Note that in this case, the
entire variable expansion must be enclosed by curly braces so that the
arguments will be associated with the instantiation of the class:

class foo(object):
    def __init__(self, arg):
        self.arg = arg

    def __call__(self, target, source, env, for_signature):
        return self.arg + " bar"

# Will expand $BAR to "my argument bar baz"
env=Environment(FOO=foo, BAR="${FOO('my argument')} baz")


On Mon, Feb 27, 2017 at 9:54 AM, Andrew C. Morrow <andrew.c.morrow at gmail.com
> wrote:

>
> Thanks Bill -
>
> One wrinkle. I defined SHLINKCOMSTR as follows:
>
> env['SHLINKCOMSTR'] = "Via tempfile: $SHLINKCOM"
>
> However, in the output, it appears that the $( $) stuff isn't being elided
> as expected:
>
> Via tempfile: /opt/mongodbtoolchain/v2/bin/clang++ -o
> build/debug/third_party/libshim_pcrecpp.so -Wl,--no-as-needed -pthread
> -Wl,-z,now -rdynamic -Wl,--fatal-warnings -fstack-protector-strong
> -fuse-ld=gold -Wl,--build-id -Wl,--hash-style=gnu
> -Wl,--detect-odr-violations -Wl,-z,noexecstack -Wl,--warn-execstack
> -Wl,-z,relro -shared   build/debug/third_party/shim_pcrecpp.os $(  $)
>  -Wl,-z,defs -Wl,--whole-archive -Wl,--start-group
> build/debug/third_party/pcre-8.39/libpcrecpp.so -Wl,--end-group
> -Wl,--no-whole-archive  -lm -lrt -ldl
>
> Is there a way to express that I want "command" (a.k.a raw=2) like
> expansion for SHLINKCOMSTR here so that they are dropped?
>
> Thanks,
> Andrew
>
>
> On Mon, Feb 27, 2017 at 11:59 AM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> Andrew,
>>
>> Indeed. One variable for command line, one for output, for exactly the
>> reason you specify.
>> You may want or/not want the same for both.
>>
>> -Bill
>>
>> On Sun, Feb 26, 2017 at 9:56 PM, Andrew C. Morrow <
>> andrew.c.morrow at gmail.com> wrote:
>>
>>>
>>> Ah, I figured out the printing, I think.
>>>
>>> Normally in a CI build I don't want to set any of the COMSTR variables,
>>> because I want the real logging of all commands. But if I have TEMPFILE in
>>> play, I *do* want to set a [SH]LINKCOMSTR, and I can just set it to print
>>> out $LINKCOM, which gets me the "real" command line. Neat.
>>>
>>>
>>>
>>> On Sun, Feb 26, 2017 at 11:22 PM, Andrew C. Morrow <
>>> andrew.c.morrow at gmail.com> wrote:
>>>
>>>>
>>>> Hi Bill -
>>>>
>>>> OK, I did try it, and.... it works! But not right out of the box.
>>>>
>>>> As it turns out, even though the generic Environment setup in
>>>> Platform/__init__.py and the POSIX Platform/posix.py does try to set this
>>>> up, the 'link.py' tool (and probably others) don't insert the TEMPFILE bit.
>>>> So, it only works on windows, because mslink.py does do it.
>>>>
>>>> I was able to work around this in my top level SConstruct file by
>>>> adding the following after all Tools were loaded:
>>>>
>>>> if not 'TEMPFILE' in env['LINKCOM']:
>>>>     env["LINKCOM"] = "${{TEMPFILE('{}', 'LINKCOMSTR')}}".format(env['L
>>>> INKCOM'])
>>>> if not 'TEMPFILE' in env['SHLINKCOM']:
>>>>     env["SHLINKCOM"] = "${{TEMPFILE('{}', 'SHLINKCOMSTR')}}".format(env[
>>>> 'SHLINKCOM'])
>>>>
>>>> I think this is probably a bug in SCons?
>>>>
>>>> One observation about this that still makes me somewhat prefer my
>>>> thin_archive approach. With the TEMPFILE approach, the actual build
>>>> invocation isn't written to the output, just a message like:
>>>>
>>>> g++ @/tmp/tmpxGAiil.lnk
>>>>
>>>> That isn't too great in a CI system, because now the actual build
>>>> command we ran isn't printed. Perhaps there is a way to get TempFileMunge
>>>> to echo the full command? As you mentioned, it isn't terribly well
>>>> documented.
>>>>
>>>> Thanks,
>>>> Andrew
>>>>
>>>>
>>>> On Mon, Oct 24, 2016 at 11:36 PM, Bill Deegan <
>>>> bill at baddogconsulting.com> wrote:
>>>>
>>>>> Did you try using TEMPFILE? and MAXLINELENGTH?
>>>>> Which I'm just seeing is far from adequately documented.
>>>>>
>>>>> see src/engine/SCons/Platform/__init__.py
>>>>>
>>>>> """A callable class.  You can set an Environment variable to this,
>>>>> then call it with a string argument, then it will perform temporary
>>>>> file substitution on it.  This is used to circumvent the long command
>>>>> line limitation.
>>>>>
>>>>> Example usage:
>>>>> env["TEMPFILE"] = TempFileMunge
>>>>> env["LINKCOM"] = "${TEMPFILE('$LINK $TARGET $SOURCES','$LINKCOMSTR')}"
>>>>>
>>>>> By default, the name of the temporary file used begins with a
>>>>> prefix of '@'.  This may be configred for other tool chains by
>>>>> setting '$TEMPFILEPREFIX'.
>>>>>
>>>>> env["TEMPFILEPREFIX"] = '-@'        # diab compiler
>>>>> env["TEMPFILEPREFIX"] = '-via'      # arm tool chain
>>>>> """
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 24, 2016 at 10:29 PM, Bill Deegan <
>>>>> bill at baddogconsulting.com> wrote:
>>>>>
>>>>>> Andrew,
>>>>>>
>>>>>> I guess the question is would the extra work be worth it in terms of
>>>>>> reduced build time?
>>>>>> Would covering all binutils platforms resolve the issue for you and
>>>>>> your team?
>>>>>> Or would you still be left with needing to use  thin archives on some
>>>>>> platforms?
>>>>>> Which other linkers are you dealing with?
>>>>>>
>>>>>> -Bill
>>>>>>
>>>>>> On Mon, Oct 24, 2016 at 4:47 PM, Andrew C. Morrow <
>>>>>> andrew.c.morrow at gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>> That would probably work on binutils platforms, but I think it would
>>>>>>> require generating the linker script from the source list, and then feeding
>>>>>>> that into the link step. Without having really investigated, that
>>>>>>> intuitively felt like it would be more work to get SCons to do that than
>>>>>>> would adding some letters to ARFLAGS and synthesizing a signature for the
>>>>>>> target.
>>>>>>>
>>>>>>> On Mon, Oct 24, 2016 at 2:30 PM, Bill Deegan <
>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>
>>>>>>>> Would something like this work?
>>>>>>>> https://cygwin.com/ml/cygwin/2004-04/msg00330.html
>>>>>>>>
>>>>>>>> Or is this for a non-gcc compiler?
>>>>>>>> -Bill
>>>>>>>>
>>>>>>>> On Mon, Oct 24, 2016 at 10:12 AM, Andrew C. Morrow <
>>>>>>>> andrew.c.morrow at gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> The short answer is that when linking by enumerating all object
>>>>>>>>> files on the command line, we recently hit the hard coded command line
>>>>>>>>> length limit on certain linux variants. By using archives to bundle groups
>>>>>>>>> of objects into logical libraries, we can sidestep that limitation. The
>>>>>>>>> downside was increased IO and disk utilization, but thin archives mitigate
>>>>>>>>> that.
>>>>>>>>>
>>>>>>>>> I'm still curious though - any thoughts on the implementation? Is
>>>>>>>>> the mechanism used for interposing on the signature calculations correct?
>>>>>>>>> Is there a better, more scons-y way?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Oct 21, 2016 at 8:05 PM, Bill Deegan <
>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>
>>>>>>>>>> I'm guessing that using archives let's him skip keeping track of
>>>>>>>>>> which object files need to be pulled in (as opposed to listing all the
>>>>>>>>>> object files).
>>>>>>>>>>
>>>>>>>>>> -Bill
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 21, 2016 at 11:26 AM, Rob Boehne <robb at datalogics.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Andrew,
>>>>>>>>>>>
>>>>>>>>>>> Just out of curiosity, why are you using archives at all?  There
>>>>>>>>>>> are legitimate reasons, and I haven’t looked at MongoDB, but perhaps
>>>>>>>>>>> removing this step would simplify your build and solve this issue.
>>>>>>>>>>>
>>>>>>>>>>> HTH,
>>>>>>>>>>>
>>>>>>>>>>> Robert Boehne
>>>>>>>>>>>
>>>>>>>>>>> From: Scons-users <scons-users-bounces at scons.org> on behalf of
>>>>>>>>>>> "Andrew C. Morrow" <andrew.c.morrow at gmail.com>
>>>>>>>>>>> Reply-To: SCons users mailing list <scons-users at scons.org>
>>>>>>>>>>> Date: Thursday, October 20, 2016 at 3:32 PM
>>>>>>>>>>> To: SCons users mailing list <scons-users at scons.org>
>>>>>>>>>>> Subject: [Scons-users] Thin archive Tool
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi -
>>>>>>>>>>>
>>>>>>>>>>> The SCons based build system for MongoDB makes heavy use of
>>>>>>>>>>> static linking. One consequence of static linking is that the space
>>>>>>>>>>> requirements are basically doubled, since each translation unit produces an
>>>>>>>>>>> object file, and then each object file is copied into an archive file.
>>>>>>>>>>> Adding CacheDir into the mix multiplies this duplication.
>>>>>>>>>>>
>>>>>>>>>>> The GNU binutils tools, however, support 'thin' archives, where
>>>>>>>>>>> the archive contents are simply a list of file references, meaning that the
>>>>>>>>>>> archive files are very small. At link time, the linker simply dereferences
>>>>>>>>>>> the listed files in each archive.
>>>>>>>>>>>
>>>>>>>>>>> To support this, we added the following Apache 2.0 licensed tool:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>>>>> _tools/thin_archive.py
>>>>>>>>>>>
>>>>>>>>>>> One subtle aspect to consider is that when using thin archives,
>>>>>>>>>>> if application X depends on libY.a, which contains z.o produced from z.c,
>>>>>>>>>>> then if z.c is changed, the built-in signature of libY.a will not change,
>>>>>>>>>>> since the reference to z.o doesn't change when the archive is rebuilt, so
>>>>>>>>>>> taking the MD5 of the file contents will yield the same result.
>>>>>>>>>>>
>>>>>>>>>>> To address this, our Tool creates a new Node subclass that
>>>>>>>>>>> overrides the get_contents and get_content_hash methods, and sets the
>>>>>>>>>>> target_factory for StaticLibrary to produce that Node subclass. The
>>>>>>>>>>> overriding behavior computes a content hash based on the content hash of
>>>>>>>>>>> the children of the new node.
>>>>>>>>>>>
>>>>>>>>>>> This all seems to work, fairly well, but I was curious if there
>>>>>>>>>>> was a more appropriate way to accomplish this. The end goal is that we want
>>>>>>>>>>> the content signature of these Nodes to be a hash of the signatures of all
>>>>>>>>>>> of the Nodes children, rather than the on-disk contents of the Node.
>>>>>>>>>>>
>>>>>>>>>>> Is there a better way to accomplish this than what we are doing
>>>>>>>>>>> here?
>>>>>>>>>>>
>>>>>>>>>>> FWIW we also use a similar technique for driving ABI change
>>>>>>>>>>> linking when doing dynamic builds:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>>>>> _tools/abilink.py
>>>>>>>>>>>
>>>>>>>>>>> Here, we would be particularly interested in arranging to
>>>>>>>>>>> interpose in such a way to absolutely minimize the number of times we must
>>>>>>>>>>> invoke abidw, as that is very expensive. Avoiding needless re-invocations
>>>>>>>>>>> of abidw as things move in and out of the CacheDir is desired.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Andrew
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Scons-users mailing list
>>>>>>>>>>> Scons-users at scons.org
>>>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Scons-users mailing list
>>>>>>>>>> Scons-users at scons.org
>>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-users mailing list
>>>>>>>>> Scons-users at scons.org
>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170227/5d7cf310/attachment-0001.html>

More information about the Scons-users mailing list