[Scons-users] Make all Program and SharedLibrary targets implicitly Precious?

Andrew C. Morrow andrew.c.morrow at gmail.com
Tue May 17 23:13:36 EDT 2016 

I'm on whatever 16.04 ships with, which looks like it is binutils 2.26, and
that does appear to be the latest stable binutils release.

On Tue, May 17, 2016 at 6:47 PM, Bill Deegan <bill at baddogconsulting.com>
wrote:

> Andrew,
>
> Dumb question probably, but are you on the latest released binutils?
>
> -Bill
>
> On Tue, May 17, 2016 at 1:27 PM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>>
>> Hi -
>>
>> Just to clarify - was it just -Wl,-fuse-ld=gold that you tried and got
>> the speedup, or the whole incremental linking tool? I personally wasn't
>> able to get incremental binaries built with gold to work at all - they just
>> crashed in early startup. I probably should file some binutils tickets...
>>
>> Thanks,
>> Andrew
>>
>>
>> On Tue, May 17, 2016 at 4:47 AM, Ale Strooisma <
>> a.strooisma at student.utwente.nl> wrote:
>>
>>> Hi Andrew,
>>>
>>> I actually needed to do LINKFLAGS='-Wl,-fuse-ld=gold' and I must say it
>>> gives an insane speed-up. I am really happy I found out about this tool!
>>>
>>> Kind regards,
>>> Ale Strooisma
>>>
>>> On 13 May 2016 at 15:47, Andrew C. Morrow <andrew.c.morrow at gmail.com>
>>> wrote:
>>>
>>>>
>>>> Hi -
>>>>
>>>> The equivalent of env.Append(LINKFLAGS=['-fuse-ld=gold']) is what
>>>> MongoDB uses to enable the gold linker:
>>>>
>>>>
>>>> https://github.com/mongodb/mongo/blob/d9a8e6a9db1bd48c2bbfb1ad18e7cb8e18eb7302/SConstruct#L2164-L2167
>>>>
>>>> You almost certainly don't want to change LINK in a C++ build, for the
>>>> reasons you found. SCons is really fairly smart about how to invoke the
>>>> linker correctly (search for SmartLink).
>>>>
>>>> Hope this helps,
>>>> Andrew
>>>>
>>>>
>>>> On Fri, May 13, 2016 at 4:56 AM, Ale Strooisma <
>>>> a.strooisma at student.utwente.nl> wrote:
>>>>
>>>>> Ah okay, that works, thanks! Silly that I did not come up with that
>>>>> myself.
>>>>> Still a few more libraries to add, though. I'm definitely going to try
>>>>> your alternative.
>>>>>
>>>>> Kind regards,
>>>>> Ale Strooisma
>>>>>
>>>>> On 13 May 2016 at 10:44, Jörg Plate <patterner at gmail.com> wrote:
>>>>>
>>>>>> The stdc++ library tends to be in non-standard locations, since it's
>>>>>> depending on the compiler.
>>>>>> Try
>>>>>>   locate "*libstdc++*so*"
>>>>>> and use the -L option with the correct directory. My g++ uses
>>>>>> /usr/lib/gcc/x86_64-linux-gnu/6/.
>>>>>>
>>>>>> Alternative:   use g++ with "-Wl,-fuse-ld=gold" and let g++ deal with
>>>>>> it.
>>>>>>
>>>>>> On Fri, May 13, 2016 at 10:30 AM, Ale Strooisma
>>>>>> <a.strooisma at student.utwente.nl> wrote:
>>>>>> > Andrew,
>>>>>> >
>>>>>> > inspired by this discussion I decided to try using gold as well. I
>>>>>> ran into
>>>>>> > a problem however, and can't seem to find a lot of documentation on
>>>>>> gold, so
>>>>>> > I hoped you could shed some light on this problem as you are
>>>>>> working with
>>>>>> > gold and scons.
>>>>>> >
>>>>>> > I changed to gold by adding LINK='ld.gold' to my environment object.
>>>>>> > However, now I get a lot of undefined reference errors during the
>>>>>> linking
>>>>>> > process - they all seem to come from the standard libraries. I
>>>>>> interpret
>>>>>> > this as gold not linking those by default, so I added stdc++ to
>>>>>> LIBS, but
>>>>>> > that resulted in "ld.gold: error: cannot find -lstdc++".
>>>>>> >
>>>>>> > Any tips for getting this to work?
>>>>>> >
>>>>>> > Kind regards,
>>>>>> > Ale Strooisma
>>>>>> >
>>>>>> >
>>>>>> > On 13 May 2016 at 00:28, Bill Deegan <bill at baddogconsulting.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Andrew,
>>>>>> >>
>>>>>> >> Looks reasonable.
>>>>>> >> What happens to the old binary if you os.remove it on one system,
>>>>>> when
>>>>>> >> it's being used on another? (NFS)
>>>>>> >> Does it crash, or is the file kept alive via inode reference by
>>>>>> running
>>>>>> >> process?
>>>>>> >>
>>>>>> >> -Bill
>>>>>> >>
>>>>>> >> On Thu, May 12, 2016 at 4:08 PM, Andrew C. Morrow
>>>>>> >> <andrew.c.morrow at gmail.com> wrote:
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> OK, I got something that sort of works for incremental linking.
>>>>>> I'm sure
>>>>>> >>> it can be improved:
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> https://github.com/acmorrow/mongo/blob/0062b8bfa0c24812438f4765fa20755fb5f6fc54/site_scons/site_tools/incremental_link.py
>>>>>> >>>
>>>>>> >>> The binaries that ld.gold produces seem terribly broken, but
>>>>>> perhaps it
>>>>>> >>> will be useful on Windows.
>>>>>> >>>
>>>>>> >>> I wonder if .Precious should have a mode where it behaves like
>>>>>> this by
>>>>>> >>> default, or if it should take an optional parameter. It seems
>>>>>> useful.
>>>>>> >>>
>>>>>> >>> On Thu, May 12, 2016 at 12:18 PM, Andrew C. Morrow
>>>>>> >>> <andrew.c.morrow at gmail.com> wrote:
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> Hi Bill -
>>>>>> >>>>
>>>>>> >>>> I like this suggestion of adding a PreAction to each executable
>>>>>> and
>>>>>> >>>> shared library, but I don't want to have to explicitly call
>>>>>> .AddPreAction on
>>>>>> >>>> each target.
>>>>>> >>>>
>>>>>> >>>> Can you suggest an easy way to achieve the same effect globally
>>>>>> for all
>>>>>> >>>> SharedLibrary and Program targets, without writing a
>>>>>> PseudoBuilder? I'm fine
>>>>>> >>>> though with reaching into the existing builders and altering
>>>>>> them.
>>>>>> >>>>
>>>>>> >>>> Thanks,
>>>>>> >>>> Andrew
>>>>>> >>>>
>>>>>> >>>> On Sat, Apr 30, 2016 at 4:38 PM, Bill Deegan <
>>>>>> bill at baddogconsulting.com>
>>>>>> >>>> wrote:
>>>>>> >>>>>
>>>>>> >>>>> Andrew,
>>>>>> >>>>>
>>>>>> >>>>> How about a pre-action to mv the existing file to another name,
>>>>>> and
>>>>>> >>>>> then copy it to original name, then incremental link the copy?
>>>>>> >>>>> That should resolve the crashing bit.
>>>>>> >>>>>
>>>>>> >>>>> lsof filename won't help if the files being used on another
>>>>>> system, but
>>>>>> >>>>> would if on current system. (to see if in use)
>>>>>> >>>>>
>>>>>> >>>>> -Bill
>>>>>> >>>>>
>>>>>> >>>>> On Sat, Apr 30, 2016 at 3:47 PM, Andrew C. Morrow
>>>>>> >>>>> <andrew.c.morrow at gmail.com> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> Definitely a possibility. MongoDB probes for compiler support
>>>>>> for
>>>>>> >>>>>> ld.gold and prefers it by default:
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> https://github.com/mongodb/mongo/blob/master/SConstruct#L2164-L2167
>>>>>> >>>>>>
>>>>>> >>>>>> The downside to using incremental linking and making dynamic
>>>>>> targets
>>>>>> >>>>>> Precious is that by overwriting the binaries you end up
>>>>>> crashing running
>>>>>> >>>>>> programs that have them mapped. That happens often enough
>>>>>> during developer
>>>>>> >>>>>> workflow that it is unfortunate.
>>>>>> >>>>>>
>>>>>> >>>>>> So, it definitely should be opt-in behavior, and a Tool sounds
>>>>>> right
>>>>>> >>>>>> for that.
>>>>>> >>>>>>
>>>>>> >>>>>> I also wonder whether such a Tool might be able to make
>>>>>> intelligent
>>>>>> >>>>>> use of gold's --incremental-base file to avoid the above
>>>>>> problem. If you
>>>>>> >>>>>> first copied the dynamic target aside under a different name,
>>>>>> then unlinked
>>>>>> >>>>>> the original, then used --incremental-base pointing to the
>>>>>> moved-aside file,
>>>>>> >>>>>> perhaps you could avoid crashing programs that have the target
>>>>>> mapped. I'm
>>>>>> >>>>>> not sure though if the cost of the file copy would eat up most
>>>>>> of the gains
>>>>>> >>>>>> from incremental linking.
>>>>>> >>>>>>
>>>>>> >>>>>> That sounds complex enough to also warrant a separate Tool.
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> On Fri, Apr 29, 2016 at 1:52 PM, Bill Deegan
>>>>>> >>>>>> <bill at baddogconsulting.com> wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Andrew,
>>>>>> >>>>>>>
>>>>>> >>>>>>> Might make sense to make a gold linker tool?
>>>>>> >>>>>>>
>>>>>> >>>>>>> -Bill
>>>>>> >>>>>>>
>>>>>> >>>>>>> On Fri, Apr 29, 2016 at 9:47 AM, Andrew C. Morrow
>>>>>> >>>>>>> <andrew.c.morrow at gmail.com> wrote:
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Thanks Bill -
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> I think that sounds promising, I'll give it a look. If I get
>>>>>> it
>>>>>> >>>>>>>> working well I'll ping the list back and describe my
>>>>>> findings.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Thanks,
>>>>>> >>>>>>>> Andrew
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> On Fri, Apr 29, 2016 at 12:10 PM, Bill Deegan
>>>>>> >>>>>>>> <bill at baddogconsulting.com> wrote:
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Andrew,
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Likely you want to stick this logic in the emitter for
>>>>>> building
>>>>>> >>>>>>>>> libraries and programs.
>>>>>> >>>>>>>>> SCons/Tool/link.py  shlib_emitter() is the current logic.
>>>>>> >>>>>>>>> The environment tracks this in:
>>>>>> >>>>>>>>> env['SHLIBEMITTER'] which is a list.
>>>>>> >>>>>>>>> So if you append to SHLIBEMITTER and mark the relevant nodes
>>>>>> >>>>>>>>> Precious()
>>>>>> >>>>>>>>> env['PROGEMITTER'] is the emitter for programs. Looks like
>>>>>> it's
>>>>>> >>>>>>>>> only defined for mslink and qt, so you can defined one on
>>>>>> non qt/mslink
>>>>>> >>>>>>>>> freely and do the same.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Hope that helps
>>>>>> >>>>>>>>> -Bill
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> On Fri, Apr 29, 2016 at 8:52 AM, William Blevins
>>>>>> >>>>>>>>> <wblevins001 at gmail.com> wrote:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> Andrew,
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I expect that you could make a Pseudo-Builder that wraps
>>>>>> Program
>>>>>> >>>>>>>>>> and Library calls, and replace the top-level calls with
>>>>>> your wrappers.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I think you can replace them like
>>>>>> env['BUILDERS']['xxxxxx'] =
>>>>>> >>>>>>>>>> Wrapper or env.AddMethod but I haven't tested this
>>>>>> explicitly.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> https://bitbucket.org/scons/scons/wiki/ToolsForFools
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> V/R,
>>>>>> >>>>>>>>>> William
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> On Fri, Apr 29, 2016 at 4:21 PM, Andrew C. Morrow
>>>>>> >>>>>>>>>> <andrew.c.morrow at gmail.com> wrote:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Hi -
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Is there a simple way to do this? I'd like to use
>>>>>> ld.gold's
>>>>>> >>>>>>>>>>> incremental linking, but I don't want to need to add
>>>>>> .Precious to every
>>>>>> >>>>>>>>>>> executable and library in my project.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Thanks,
>>>>>> >>>>>>>>>>> Andrew
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> _______________________________________________
>>>>>> >>>>>>>>>>> Scons-users mailing list
>>>>>> >>>>>>>>>>> Scons-users at scons.org
>>>>>> >>>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> _______________________________________________
>>>>>> >>>>>>>>>> Scons-users mailing list
>>>>>> >>>>>>>>>> Scons-users at scons.org
>>>>>> >>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> _______________________________________________
>>>>>> >>>>>>>>> Scons-users mailing list
>>>>>> >>>>>>>>> Scons-users at scons.org
>>>>>> >>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> _______________________________________________
>>>>>> >>>>>>>> Scons-users mailing list
>>>>>> >>>>>>>> Scons-users at scons.org
>>>>>> >>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >>>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>> _______________________________________________
>>>>>> >>>>>>> Scons-users mailing list
>>>>>> >>>>>>> Scons-users at scons.org
>>>>>> >>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >>>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> _______________________________________________
>>>>>> >>>>>> Scons-users mailing list
>>>>>> >>>>>> Scons-users at scons.org
>>>>>> >>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >>>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> _______________________________________________
>>>>>> >>>>> Scons-users mailing list
>>>>>> >>>>> Scons-users at scons.org
>>>>>> >>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >>>>>
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> _______________________________________________
>>>>>> >>> Scons-users mailing list
>>>>>> >>> Scons-users at scons.org
>>>>>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >> _______________________________________________
>>>>>> >> Scons-users mailing list
>>>>>> >> Scons-users at scons.org
>>>>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Scons-users mailing list
>>>>>> > Scons-users at scons.org
>>>>>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> "I'm working on it."
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160517/8b1980b9/attachment-0001.html>

More information about the Scons-users mailing list