[Scons-users] Make all Program and SharedLibrary targets implicitly Precious?

[Ale Strooisma]

Hi Andrew,

I had a massive speed-up with just -Wl,-fuse-ld=gold
I don't even know what incremental linking is - haven't looked up yet.
With gold the linking went from (estimated) 5-10 seconds to nearly instant.

On 18 May 2016 at 05:13, Andrew C. Morrow <andrew.c.morrow at gmail.com> wrote:

>
> I'm on whatever 16.04 ships with, which looks like it is binutils 2.26,
> and that does appear to be the latest stable binutils release.
>
> On Tue, May 17, 2016 at 6:47 PM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> Andrew,
>>
>> Dumb question probably, but are you on the latest released binutils?
>>
>> -Bill
>>
>> On Tue, May 17, 2016 at 1:27 PM, Andrew C. Morrow <
>> andrew.c.morrow at gmail.com> wrote:
>>
>>>
>>> Hi -
>>>
>>> Just to clarify - was it just -Wl,-fuse-ld=gold that you tried and got
>>> the speedup, or the whole incremental linking tool? I personally wasn't
>>> able to get incremental binaries built with gold to work at all - they just
>>> crashed in early startup. I probably should file some binutils tickets...
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
>>> On Tue, May 17, 2016 at 4:47 AM, Ale Strooisma <
>>> a.strooisma at student.utwente.nl> wrote:
>>>
>>>> Hi Andrew,
>>>>
>>>> I actually needed to do LINKFLAGS='-Wl,-fuse-ld=gold' and I must say it
>>>> gives an insane speed-up. I am really happy I found out about this tool!
>>>>
>>>> Kind regards,
>>>> Ale Strooisma
>>>>
>>>> On 13 May 2016 at 15:47, Andrew C. Morrow <andrew.c.morrow at gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Hi -
>>>>>
>>>>> The equivalent of env.Append(LINKFLAGS=['-fuse-ld=gold']) is what
>>>>> MongoDB uses to enable the gold linker:
>>>>>
>>>>>
>>>>> https://github.com/mongodb/mongo/blob/d9a8e6a9db1bd48c2bbfb1ad18e7cb8e18eb7302/SConstruct#L2164-L2167
>>>>>
>>>>> You almost certainly don't want to change LINK in a C++ build, for the
>>>>> reasons you found. SCons is really fairly smart about how to invoke the
>>>>> linker correctly (search for SmartLink).
>>>>>
>>>>> Hope this helps,
>>>>> Andrew
>>>>>
>>>>>
>>>>> On Fri, May 13, 2016 at 4:56 AM, Ale Strooisma <
>>>>> a.strooisma at student.utwente.nl> wrote:
>>>>>
>>>>>> Ah okay, that works, thanks! Silly that I did not come up with that
>>>>>> myself.
>>>>>> Still a few more libraries to add, though. I'm definitely going to
>>>>>> try your alternative.
>>>>>>
>>>>>> Kind regards,
>>>>>> Ale Strooisma
>>>>>>
>>>>>> On 13 May 2016 at 10:44, Jörg Plate <patterner at gmail.com> wrote:
>>>>>>
>>>>>>> The stdc++ library tends to be in non-standard locations, since it's
>>>>>>> depending on the compiler.
>>>>>>> Try
>>>>>>>   locate "*libstdc++*so*"
>>>>>>> and use the -L option with the correct directory. My g++ uses
>>>>>>> /usr/lib/gcc/x86_64-linux-gnu/6/.
>>>>>>>
>>>>>>> Alternative:   use g++ with "-Wl,-fuse-ld=gold" and let g++ deal
>>>>>>> with it.
>>>>>>>
>>>>>>> On Fri, May 13, 2016 at 10:30 AM, Ale Strooisma
>>>>>>> <a.strooisma at student.utwente.nl> wrote:
>>>>>>> > Andrew,
>>>>>>> >
>>>>>>> > inspired by this discussion I decided to try using gold as well. I
>>>>>>> ran into
>>>>>>> > a problem however, and can't seem to find a lot of documentation
>>>>>>> on gold, so
>>>>>>> > I hoped you could shed some light on this problem as you are
>>>>>>> working with
>>>>>>> > gold and scons.
>>>>>>> >
>>>>>>> > I changed to gold by adding LINK='ld.gold' to my environment
>>>>>>> object.
>>>>>>> > However, now I get a lot of undefined reference errors during the
>>>>>>> linking
>>>>>>> > process - they all seem to come from the standard libraries. I
>>>>>>> interpret
>>>>>>> > this as gold not linking those by default, so I added stdc++ to
>>>>>>> LIBS, but
>>>>>>> > that resulted in "ld.gold: error: cannot find -lstdc++".
>>>>>>> >
>>>>>>> > Any tips for getting this to work?
>>>>>>> >
>>>>>>> > Kind regards,
>>>>>>> > Ale Strooisma
>>>>>>> >
>>>>>>> >
>>>>>>> > On 13 May 2016 at 00:28, Bill Deegan <bill at baddogconsulting.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Andrew,
>>>>>>> >>
>>>>>>> >> Looks reasonable.
>>>>>>> >> What happens to the old binary if you os.remove it on one system,
>>>>>>> when
>>>>>>> >> it's being used on another? (NFS)
>>>>>>> >> Does it crash, or is the file kept alive via inode reference by
>>>>>>> running
>>>>>>> >> process?
>>>>>>> >>
>>>>>>> >> -Bill
>>>>>>> >>
>>>>>>> >> On Thu, May 12, 2016 at 4:08 PM, Andrew C. Morrow
>>>>>>> >> <andrew.c.morrow at gmail.com> wrote:
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> OK, I got something that sort of works for incremental linking.
>>>>>>> I'm sure
>>>>>>> >>> it can be improved:
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> https://github.com/acmorrow/mongo/blob/0062b8bfa0c24812438f4765fa20755fb5f6fc54/site_scons/site_tools/incremental_link.py
>>>>>>> >>>
>>>>>>> >>> The binaries that ld.gold produces seem terribly broken, but
>>>>>>> perhaps it
>>>>>>> >>> will be useful on Windows.
>>>>>>> >>>
>>>>>>> >>> I wonder if .Precious should have a mode where it behaves like
>>>>>>> this by
>>>>>>> >>> default, or if it should take an optional parameter. It seems
>>>>>>> useful.
>>>>>>> >>>
>>>>>>> >>> On Thu, May 12, 2016 at 12:18 PM, Andrew C. Morrow
>>>>>>> >>> <andrew.c.morrow at gmail.com> wrote:
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>> Hi Bill -
>>>>>>> >>>>
>>>>>>> >>>> I like this suggestion of adding a PreAction to each executable
>>>>>>> and
>>>>>>> >>>> shared library, but I don't want to have to explicitly call
>>>>>>> .AddPreAction on
>>>>>>> >>>> each target.
>>>>>>> >>>>
>>>>>>> >>>> Can you suggest an easy way to achieve the same effect globally
>>>>>>> for all
>>>>>>> >>>> SharedLibrary and Program targets, without writing a
>>>>>>> PseudoBuilder? I'm fine
>>>>>>> >>>> though with reaching into the existing builders and altering
>>>>>>> them.
>>>>>>> >>>>
>>>>>>> >>>> Thanks,
>>>>>>> >>>> Andrew
>>>>>>> >>>>
>>>>>>> >>>> On Sat, Apr 30, 2016 at 4:38 PM, Bill Deegan <
>>>>>>> bill at baddogconsulting.com>
>>>>>>> >>>> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> Andrew,
>>>>>>> >>>>>
>>>>>>> >>>>> How about a pre-action to mv the existing file to another
>>>>>>> name, and
>>>>>>> >>>>> then copy it to original name, then incremental link the copy?
>>>>>>> >>>>> That should resolve the crashing bit.
>>>>>>> >>>>>
>>>>>>> >>>>> lsof filename won't help if the files being used on another
>>>>>>> system, but
>>>>>>> >>>>> would if on current system. (to see if in use)
>>>>>>> >>>>>
>>>>>>> >>>>> -Bill
>>>>>>> >>>>>
>>>>>>> >>>>> On Sat, Apr 30, 2016 at 3:47 PM, Andrew C. Morrow
>>>>>>> >>>>> <andrew.c.morrow at gmail.com> wrote:
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> >>>>>> Definitely a possibility. MongoDB probes for compiler support
>>>>>>> for
>>>>>>> >>>>>> ld.gold and prefers it by default:
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> https://github.com/mongodb/mongo/blob/master/SConstruct#L2164-L2167
>>>>>>> >>>>>>
>>>>>>> >>>>>> The downside to using incremental linking and making dynamic
>>>>>>> targets
>>>>>>> >>>>>> Precious is that by overwriting the binaries you end up
>>>>>>> crashing running
>>>>>>> >>>>>> programs that have them mapped. That happens often enough
>>>>>>> during developer
>>>>>>> >>>>>> workflow that it is unfortunate.
>>>>>>> >>>>>>
>>>>>>> >>>>>> So, it definitely should be opt-in behavior, and a Tool
>>>>>>> sounds right
>>>>>>> >>>>>> for that.
>>>>>>> >>>>>>
>>>>>>> >>>>>> I also wonder whether such a Tool might be able to make
>>>>>>> intelligent
>>>>>>> >>>>>> use of gold's --incremental-base file to avoid the above
>>>>>>> problem. If you
>>>>>>> >>>>>> first copied the dynamic target aside under a different name,
>>>>>>> then unlinked
>>>>>>> >>>>>> the original, then used --incremental-base pointing to the
>>>>>>> moved-aside file,
>>>>>>> >>>>>> perhaps you could avoid crashing programs that have the
>>>>>>> target mapped. I'm
>>>>>>> >>>>>> not sure though if the cost of the file copy would eat up
>>>>>>> most of the gains
>>>>>>> >>>>>> from incremental linking.
>>>>>>> >>>>>>
>>>>>>> >>>>>> That sounds complex enough to also warrant a separate Tool.
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> >>>>>> On Fri, Apr 29, 2016 at 1:52 PM, Bill Deegan
>>>>>>> >>>>>> <bill at baddogconsulting.com> wrote:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Andrew,
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Might make sense to make a gold linker tool?
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> -Bill
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> On Fri, Apr 29, 2016 at 9:47 AM, Andrew C. Morrow
>>>>>>> >>>>>>> <andrew.c.morrow at gmail.com> wrote:
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> Thanks Bill -
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> I think that sounds promising, I'll give it a look. If I
>>>>>>> get it
>>>>>>> >>>>>>>> working well I'll ping the list back and describe my
>>>>>>> findings.
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> Thanks,
>>>>>>> >>>>>>>> Andrew
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> On Fri, Apr 29, 2016 at 12:10 PM, Bill Deegan
>>>>>>> >>>>>>>> <bill at baddogconsulting.com> wrote:
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> Andrew,
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> Likely you want to stick this logic in the emitter for
>>>>>>> building
>>>>>>> >>>>>>>>> libraries and programs.
>>>>>>> >>>>>>>>> SCons/Tool/link.py  shlib_emitter() is the current logic.
>>>>>>> >>>>>>>>> The environment tracks this in:
>>>>>>> >>>>>>>>> env['SHLIBEMITTER'] which is a list.
>>>>>>> >>>>>>>>> So if you append to SHLIBEMITTER and mark the relevant
>>>>>>> nodes
>>>>>>> >>>>>>>>> Precious()
>>>>>>> >>>>>>>>> env['PROGEMITTER'] is the emitter for programs. Looks like
>>>>>>> it's
>>>>>>> >>>>>>>>> only defined for mslink and qt, so you can defined one on
>>>>>>> non qt/mslink
>>>>>>> >>>>>>>>> freely and do the same.
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> Hope that helps
>>>>>>> >>>>>>>>> -Bill
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> On Fri, Apr 29, 2016 at 8:52 AM, William Blevins
>>>>>>> >>>>>>>>> <wblevins001 at gmail.com> wrote:
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> Andrew,
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> I expect that you could make a Pseudo-Builder that wraps
>>>>>>> Program
>>>>>>> >>>>>>>>>> and Library calls, and replace the top-level calls with
>>>>>>> your wrappers.
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> I think you can replace them like
>>>>>>> env['BUILDERS']['xxxxxx'] =
>>>>>>> >>>>>>>>>> Wrapper or env.AddMethod but I haven't tested this
>>>>>>> explicitly.
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> https://bitbucket.org/scons/scons/wiki/ToolsForFools
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> V/R,
>>>>>>> >>>>>>>>>> William
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> On Fri, Apr 29, 2016 at 4:21 PM, Andrew C. Morrow
>>>>>>> >>>>>>>>>> <andrew.c.morrow at gmail.com> wrote:
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>> Hi -
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>> Is there a simple way to do this? I'd like to use
>>>>>>> ld.gold's
>>>>>>> >>>>>>>>>>> incremental linking, but I don't want to need to add
>>>>>>> .Precious to every
>>>>>>> >>>>>>>>>>> executable and library in my project.
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>> >>>>>>>>>>> Andrew
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>> _______________________________________________
>>>>>>> >>>>>>>>>>> Scons-users mailing list
>>>>>>> >>>>>>>>>>> Scons-users at scons.org
>>>>>>> >>>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> _______________________________________________
>>>>>>> >>>>>>>>>> Scons-users mailing list
>>>>>>> >>>>>>>>>> Scons-users at scons.org
>>>>>>> >>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> _______________________________________________
>>>>>>> >>>>>>>>> Scons-users mailing list
>>>>>>> >>>>>>>>> Scons-users at scons.org
>>>>>>> >>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> _______________________________________________
>>>>>>> >>>>>>>> Scons-users mailing list
>>>>>>> >>>>>>>> Scons-users at scons.org
>>>>>>> >>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> _______________________________________________
>>>>>>> >>>>>>> Scons-users mailing list
>>>>>>> >>>>>>> Scons-users at scons.org
>>>>>>> >>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >>>>>>>
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> >>>>>> _______________________________________________
>>>>>>> >>>>>> Scons-users mailing list
>>>>>>> >>>>>> Scons-users at scons.org
>>>>>>> >>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >>>>>>
>>>>>>> >>>>>
>>>>>>> >>>>>
>>>>>>> >>>>> _______________________________________________
>>>>>>> >>>>> Scons-users mailing list
>>>>>>> >>>>> Scons-users at scons.org
>>>>>>> >>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >>>>>
>>>>>>> >>>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> _______________________________________________
>>>>>>> >>> Scons-users mailing list
>>>>>>> >>> Scons-users at scons.org
>>>>>>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >>>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> _______________________________________________
>>>>>>> >> Scons-users mailing list
>>>>>>> >> Scons-users at scons.org
>>>>>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >>
>>>>>>> >
>>>>>>> >
>>>>>>> > _______________________________________________
>>>>>>> > Scons-users mailing list
>>>>>>> > Scons-users at scons.org
>>>>>>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> "I'm working on it."
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160518/fd7b1838/attachment-0001.html>