[Scons-users] Thin archive Tool

Sat Jan 7 17:33:26 EST 2017

Library dependencies are explicit only to my knowledge. Scanners are only
for implicit dependencies.

V/R,
William

On Sat, Jan 7, 2017 at 3:44 PM, Andrew C. Morrow <andrew.c.morrow at gmail.com>
wrote:

>
> Thanks Bill.
>
> To clarify, I'm in the case of both tools, I'm only applying this override
> for either static archive nodes (.a) or dynamic library nodes (.so, .dylib,
> etc.). Are there by default any built-int SCons scanners that apply to such
> nodes? As binary artifacts, I suspect the answer is "no".
>
> Thanks,
> Andrew
>
>
> On Wed, Jan 4, 2017 at 11:57 AM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> off the top of my head..
>> get_content is used by scanners so overriding that could impact those.
>>
>> I'll have to re-digest the email thread to have smarter response..
>> Probably later today.
>>
>> -Bill
>>
>>
>> On Wed, Jan 4, 2017 at 6:38 AM, Andrew C. Morrow <
>> andrew.c.morrow at gmail.com> wrote:
>>
>>>
>>> Hi -
>>>
>>> I'm circling back to this issue for various reasons, but I'd like to
>>> sidestep the questions about whether thin archives are a good idea or not,
>>> or whether temp files are a solution. What I am more interested in is
>>> determining the correct way to override the signature calculation for a
>>> file node.
>>>
>>> One approach, which I'm now using in two custom Tools, is to override
>>> the get_content and get_content_hash methods of FS.File. The alternative
>>> would be to directly override FS.File.get_csig, and leave get_content and
>>> get_content_hash as they are.
>>>
>>> The advantage to overriding get_content and get_content_hash is that
>>> they are simple methods, and by overriding them, the existing
>>> FS.File.get_csig mechanics just do the right thing. But I'm somewhat
>>> worried that there may be other parts of SCons outside the signature
>>> calculation mechanism that might want the actual file contents for some
>>> reason, not the "fake contents" I'm returning to compute a signature for
>>> the node.
>>>
>>> Overriding FS.File.get_csig seems like a more targeted approach, and so
>>> less likely to interfere if other parts of SCons really do need file
>>> contents, but it would mean that I would need to re-create the logic of
>>> FS.File.get_csig
>>> <http://scons.org/doc/HTML/scons-api/SCons.Node.FS-pysrc.html#File.get_csig>
>>> multiple times, and that seems likely to drift out of sync with SCons
>>> itself.
>>>
>>> Any guidance?
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
>>> On Mon, Oct 24, 2016 at 11:36 PM, Bill Deegan <bill at baddogconsulting.com
>>> > wrote:
>>>
>>>> Did you try using TEMPFILE? and MAXLINELENGTH?
>>>> Which I'm just seeing is far from adequately documented.
>>>>
>>>> see src/engine/SCons/Platform/__init__.py
>>>>
>>>> """A callable class.  You can set an Environment variable to this,
>>>> then call it with a string argument, then it will perform temporary
>>>> file substitution on it.  This is used to circumvent the long command
>>>> line limitation.
>>>>
>>>> Example usage:
>>>> env["TEMPFILE"] = TempFileMunge
>>>> env["LINKCOM"] = "${TEMPFILE('$LINK $TARGET $SOURCES','$LINKCOMSTR')}"
>>>>
>>>> By default, the name of the temporary file used begins with a
>>>> prefix of '@'.  This may be configred for other tool chains by
>>>> setting '$TEMPFILEPREFIX'.
>>>>
>>>> env["TEMPFILEPREFIX"] = '-@'        # diab compiler
>>>> env["TEMPFILEPREFIX"] = '-via'      # arm tool chain
>>>> """
>>>>
>>>>
>>>>
>>>> On Mon, Oct 24, 2016 at 10:29 PM, Bill Deegan <
>>>> bill at baddogconsulting.com> wrote:
>>>>
>>>>> Andrew,
>>>>>
>>>>> I guess the question is would the extra work be worth it in terms of
>>>>> reduced build time?
>>>>> Would covering all binutils platforms resolve the issue for you and
>>>>> your team?
>>>>> Or would you still be left with needing to use  thin archives on some
>>>>> platforms?
>>>>> Which other linkers are you dealing with?
>>>>>
>>>>> -Bill
>>>>>
>>>>> On Mon, Oct 24, 2016 at 4:47 PM, Andrew C. Morrow <
>>>>> andrew.c.morrow at gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> That would probably work on binutils platforms, but I think it would
>>>>>> require generating the linker script from the source list, and then feeding
>>>>>> that into the link step. Without having really investigated, that
>>>>>> intuitively felt like it would be more work to get SCons to do that than
>>>>>> would adding some letters to ARFLAGS and synthesizing a signature for the
>>>>>> target.
>>>>>>
>>>>>> On Mon, Oct 24, 2016 at 2:30 PM, Bill Deegan <
>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>
>>>>>>> Would something like this work?
>>>>>>> https://cygwin.com/ml/cygwin/2004-04/msg00330.html
>>>>>>>
>>>>>>> Or is this for a non-gcc compiler?
>>>>>>> -Bill
>>>>>>>
>>>>>>> On Mon, Oct 24, 2016 at 10:12 AM, Andrew C. Morrow <
>>>>>>> andrew.c.morrow at gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> The short answer is that when linking by enumerating all object
>>>>>>>> files on the command line, we recently hit the hard coded command line
>>>>>>>> length limit on certain linux variants. By using archives to bundle groups
>>>>>>>> of objects into logical libraries, we can sidestep that limitation. The
>>>>>>>> downside was increased IO and disk utilization, but thin archives mitigate
>>>>>>>> that.
>>>>>>>>
>>>>>>>> I'm still curious though - any thoughts on the implementation? Is
>>>>>>>> the mechanism used for interposing on the signature calculations correct?
>>>>>>>> Is there a better, more scons-y way?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 21, 2016 at 8:05 PM, Bill Deegan <
>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>
>>>>>>>>> I'm guessing that using archives let's him skip keeping track of
>>>>>>>>> which object files need to be pulled in (as opposed to listing all the
>>>>>>>>> object files).
>>>>>>>>>
>>>>>>>>> -Bill
>>>>>>>>>
>>>>>>>>> On Fri, Oct 21, 2016 at 11:26 AM, Rob Boehne <robb at datalogics.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Andrew,
>>>>>>>>>>
>>>>>>>>>> Just out of curiosity, why are you using archives at all?  There
>>>>>>>>>> are legitimate reasons, and I haven’t looked at MongoDB, but perhaps
>>>>>>>>>> removing this step would simplify your build and solve this issue.
>>>>>>>>>>
>>>>>>>>>> HTH,
>>>>>>>>>>
>>>>>>>>>> Robert Boehne
>>>>>>>>>>
>>>>>>>>>> From: Scons-users <scons-users-bounces at scons.org> on behalf of
>>>>>>>>>> "Andrew C. Morrow" <andrew.c.morrow at gmail.com>
>>>>>>>>>> Reply-To: SCons users mailing list <scons-users at scons.org>
>>>>>>>>>> Date: Thursday, October 20, 2016 at 3:32 PM
>>>>>>>>>> To: SCons users mailing list <scons-users at scons.org>
>>>>>>>>>> Subject: [Scons-users] Thin archive Tool
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi -
>>>>>>>>>>
>>>>>>>>>> The SCons based build system for MongoDB makes heavy use of
>>>>>>>>>> static linking. One consequence of static linking is that the space
>>>>>>>>>> requirements are basically doubled, since each translation unit produces an
>>>>>>>>>> object file, and then each object file is copied into an archive file.
>>>>>>>>>> Adding CacheDir into the mix multiplies this duplication.
>>>>>>>>>>
>>>>>>>>>> The GNU binutils tools, however, support 'thin' archives, where
>>>>>>>>>> the archive contents are simply a list of file references, meaning that the
>>>>>>>>>> archive files are very small. At link time, the linker simply dereferences
>>>>>>>>>> the listed files in each archive.
>>>>>>>>>>
>>>>>>>>>> To support this, we added the following Apache 2.0 licensed tool:
>>>>>>>>>>
>>>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>>>> _tools/thin_archive.py
>>>>>>>>>>
>>>>>>>>>> One subtle aspect to consider is that when using thin archives,
>>>>>>>>>> if application X depends on libY.a, which contains z.o produced from z.c,
>>>>>>>>>> then if z.c is changed, the built-in signature of libY.a will not change,
>>>>>>>>>> since the reference to z.o doesn't change when the archive is rebuilt, so
>>>>>>>>>> taking the MD5 of the file contents will yield the same result.
>>>>>>>>>>
>>>>>>>>>> To address this, our Tool creates a new Node subclass that
>>>>>>>>>> overrides the get_contents and get_content_hash methods, and sets the
>>>>>>>>>> target_factory for StaticLibrary to produce that Node subclass. The
>>>>>>>>>> overriding behavior computes a content hash based on the content hash of
>>>>>>>>>> the children of the new node.
>>>>>>>>>>
>>>>>>>>>> This all seems to work, fairly well, but I was curious if there
>>>>>>>>>> was a more appropriate way to accomplish this. The end goal is that we want
>>>>>>>>>> the content signature of these Nodes to be a hash of the signatures of all
>>>>>>>>>> of the Nodes children, rather than the on-disk contents of the Node.
>>>>>>>>>>
>>>>>>>>>> Is there a better way to accomplish this than what we are doing
>>>>>>>>>> here?
>>>>>>>>>>
>>>>>>>>>> FWIW we also use a similar technique for driving ABI change
>>>>>>>>>> linking when doing dynamic builds:
>>>>>>>>>>
>>>>>>>>>> https://github.com/mongodb/mongo/blob/master/site_scons/site
>>>>>>>>>> _tools/abilink.py
>>>>>>>>>>
>>>>>>>>>> Here, we would be particularly interested in arranging to
>>>>>>>>>> interpose in such a way to absolutely minimize the number of times we must
>>>>>>>>>> invoke abidw, as that is very expensive. Avoiding needless re-invocations
>>>>>>>>>> of abidw as things move in and out of the CacheDir is desired.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Andrew
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Scons-users mailing list
>>>>>>>>>> Scons-users at scons.org
>>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-users mailing list
>>>>>>>>> Scons-users at scons.org
>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170107/37434847/attachment-0001.html>