[Scons-users] Make all Program and SharedLibrary targets implicitly Precious?

Tue May 17 18:47:34 EDT 2016

Andrew,

Dumb question probably, but are you on the latest released binutils?

-Bill

On Tue, May 17, 2016 at 1:27 PM, Andrew C. Morrow <andrew.c.morrow at gmail.com
> wrote:

>
> Hi -
>
> Just to clarify - was it just -Wl,-fuse-ld=gold that you tried and got the
> speedup, or the whole incremental linking tool? I personally wasn't able to
> get incremental binaries built with gold to work at all - they just crashed
> in early startup. I probably should file some binutils tickets...
>
> Thanks,
> Andrew
>
>
> On Tue, May 17, 2016 at 4:47 AM, Ale Strooisma <
> a.strooisma at student.utwente.nl> wrote:
>
>> Hi Andrew,
>>
>> I actually needed to do LINKFLAGS='-Wl,-fuse-ld=gold' and I must say it
>> gives an insane speed-up. I am really happy I found out about this tool!
>>
>> Kind regards,
>> Ale Strooisma
>>
>> On 13 May 2016 at 15:47, Andrew C. Morrow <andrew.c.morrow at gmail.com>
>> wrote:
>>
>>>
>>> Hi -
>>>
>>> The equivalent of env.Append(LINKFLAGS=['-fuse-ld=gold']) is what
>>> MongoDB uses to enable the gold linker:
>>>
>>>
>>> https://github.com/mongodb/mongo/blob/d9a8e6a9db1bd48c2bbfb1ad18e7cb8e18eb7302/SConstruct#L2164-L2167
>>>
>>> You almost certainly don't want to change LINK in a C++ build, for the
>>> reasons you found. SCons is really fairly smart about how to invoke the
>>> linker correctly (search for SmartLink).
>>>
>>> Hope this helps,
>>> Andrew
>>>
>>>
>>> On Fri, May 13, 2016 at 4:56 AM, Ale Strooisma <
>>> a.strooisma at student.utwente.nl> wrote:
>>>
>>>> Ah okay, that works, thanks! Silly that I did not come up with that
>>>> myself.
>>>> Still a few more libraries to add, though. I'm definitely going to try
>>>> your alternative.
>>>>
>>>> Kind regards,
>>>> Ale Strooisma
>>>>
>>>> On 13 May 2016 at 10:44, Jörg Plate <patterner at gmail.com> wrote:
>>>>
>>>>> The stdc++ library tends to be in non-standard locations, since it's
>>>>> depending on the compiler.
>>>>> Try
>>>>>   locate "*libstdc++*so*"
>>>>> and use the -L option with the correct directory. My g++ uses
>>>>> /usr/lib/gcc/x86_64-linux-gnu/6/.
>>>>>
>>>>> Alternative:   use g++ with "-Wl,-fuse-ld=gold" and let g++ deal with
>>>>> it.
>>>>>
>>>>> On Fri, May 13, 2016 at 10:30 AM, Ale Strooisma
>>>>> <a.strooisma at student.utwente.nl> wrote:
>>>>> > Andrew,
>>>>> >
>>>>> > inspired by this discussion I decided to try using gold as well. I
>>>>> ran into
>>>>> > a problem however, and can't seem to find a lot of documentation on
>>>>> gold, so
>>>>> > I hoped you could shed some light on this problem as you are working
>>>>> with
>>>>> > gold and scons.
>>>>> >
>>>>> > I changed to gold by adding LINK='ld.gold' to my environment object.
>>>>> > However, now I get a lot of undefined reference errors during the
>>>>> linking
>>>>> > process - they all seem to come from the standard libraries. I
>>>>> interpret
>>>>> > this as gold not linking those by default, so I added stdc++ to
>>>>> LIBS, but
>>>>> > that resulted in "ld.gold: error: cannot find -lstdc++".
>>>>> >
>>>>> > Any tips for getting this to work?
>>>>> >
>>>>> > Kind regards,
>>>>> > Ale Strooisma
>>>>> >
>>>>> >
>>>>> > On 13 May 2016 at 00:28, Bill Deegan <bill at baddogconsulting.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Andrew,
>>>>> >>
>>>>> >> Looks reasonable.
>>>>> >> What happens to the old binary if you os.remove it on one system,
>>>>> when
>>>>> >> it's being used on another? (NFS)
>>>>> >> Does it crash, or is the file kept alive via inode reference by
>>>>> running
>>>>> >> process?
>>>>> >>
>>>>> >> -Bill
>>>>> >>
>>>>> >> On Thu, May 12, 2016 at 4:08 PM, Andrew C. Morrow
>>>>> >> <andrew.c.morrow at gmail.com> wrote:
>>>>> >>>
>>>>> >>>
>>>>> >>> OK, I got something that sort of works for incremental linking.
>>>>> I'm sure
>>>>> >>> it can be improved:
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> https://github.com/acmorrow/mongo/blob/0062b8bfa0c24812438f4765fa20755fb5f6fc54/site_scons/site_tools/incremental_link.py
>>>>> >>>
>>>>> >>> The binaries that ld.gold produces seem terribly broken, but
>>>>> perhaps it
>>>>> >>> will be useful on Windows.
>>>>> >>>
>>>>> >>> I wonder if .Precious should have a mode where it behaves like
>>>>> this by
>>>>> >>> default, or if it should take an optional parameter. It seems
>>>>> useful.
>>>>> >>>
>>>>> >>> On Thu, May 12, 2016 at 12:18 PM, Andrew C. Morrow
>>>>> >>> <andrew.c.morrow at gmail.com> wrote:
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> Hi Bill -
>>>>> >>>>
>>>>> >>>> I like this suggestion of adding a PreAction to each executable
>>>>> and
>>>>> >>>> shared library, but I don't want to have to explicitly call
>>>>> .AddPreAction on
>>>>> >>>> each target.
>>>>> >>>>
>>>>> >>>> Can you suggest an easy way to achieve the same effect globally
>>>>> for all
>>>>> >>>> SharedLibrary and Program targets, without writing a
>>>>> PseudoBuilder? I'm fine
>>>>> >>>> though with reaching into the existing builders and altering them.
>>>>> >>>>
>>>>> >>>> Thanks,
>>>>> >>>> Andrew
>>>>> >>>>
>>>>> >>>> On Sat, Apr 30, 2016 at 4:38 PM, Bill Deegan <
>>>>> bill at baddogconsulting.com>
>>>>> >>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Andrew,
>>>>> >>>>>
>>>>> >>>>> How about a pre-action to mv the existing file to another name,
>>>>> and
>>>>> >>>>> then copy it to original name, then incremental link the copy?
>>>>> >>>>> That should resolve the crashing bit.
>>>>> >>>>>
>>>>> >>>>> lsof filename won't help if the files being used on another
>>>>> system, but
>>>>> >>>>> would if on current system. (to see if in use)
>>>>> >>>>>
>>>>> >>>>> -Bill
>>>>> >>>>>
>>>>> >>>>> On Sat, Apr 30, 2016 at 3:47 PM, Andrew C. Morrow
>>>>> >>>>> <andrew.c.morrow at gmail.com> wrote:
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> Definitely a possibility. MongoDB probes for compiler support
>>>>> for
>>>>> >>>>>> ld.gold and prefers it by default:
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> https://github.com/mongodb/mongo/blob/master/SConstruct#L2164-L2167
>>>>> >>>>>>
>>>>> >>>>>> The downside to using incremental linking and making dynamic
>>>>> targets
>>>>> >>>>>> Precious is that by overwriting the binaries you end up
>>>>> crashing running
>>>>> >>>>>> programs that have them mapped. That happens often enough
>>>>> during developer
>>>>> >>>>>> workflow that it is unfortunate.
>>>>> >>>>>>
>>>>> >>>>>> So, it definitely should be opt-in behavior, and a Tool sounds
>>>>> right
>>>>> >>>>>> for that.
>>>>> >>>>>>
>>>>> >>>>>> I also wonder whether such a Tool might be able to make
>>>>> intelligent
>>>>> >>>>>> use of gold's --incremental-base file to avoid the above
>>>>> problem. If you
>>>>> >>>>>> first copied the dynamic target aside under a different name,
>>>>> then unlinked
>>>>> >>>>>> the original, then used --incremental-base pointing to the
>>>>> moved-aside file,
>>>>> >>>>>> perhaps you could avoid crashing programs that have the target
>>>>> mapped. I'm
>>>>> >>>>>> not sure though if the cost of the file copy would eat up most
>>>>> of the gains
>>>>> >>>>>> from incremental linking.
>>>>> >>>>>>
>>>>> >>>>>> That sounds complex enough to also warrant a separate Tool.
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> On Fri, Apr 29, 2016 at 1:52 PM, Bill Deegan
>>>>> >>>>>> <bill at baddogconsulting.com> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> Andrew,
>>>>> >>>>>>>
>>>>> >>>>>>> Might make sense to make a gold linker tool?
>>>>> >>>>>>>
>>>>> >>>>>>> -Bill
>>>>> >>>>>>>
>>>>> >>>>>>> On Fri, Apr 29, 2016 at 9:47 AM, Andrew C. Morrow
>>>>> >>>>>>> <andrew.c.morrow at gmail.com> wrote:
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> Thanks Bill -
>>>>> >>>>>>>>
>>>>> >>>>>>>> I think that sounds promising, I'll give it a look. If I get
>>>>> it
>>>>> >>>>>>>> working well I'll ping the list back and describe my findings.
>>>>> >>>>>>>>
>>>>> >>>>>>>> Thanks,
>>>>> >>>>>>>> Andrew
>>>>> >>>>>>>>
>>>>> >>>>>>>> On Fri, Apr 29, 2016 at 12:10 PM, Bill Deegan
>>>>> >>>>>>>> <bill at baddogconsulting.com> wrote:
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Andrew,
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Likely you want to stick this logic in the emitter for
>>>>> building
>>>>> >>>>>>>>> libraries and programs.
>>>>> >>>>>>>>> SCons/Tool/link.py  shlib_emitter() is the current logic.
>>>>> >>>>>>>>> The environment tracks this in:
>>>>> >>>>>>>>> env['SHLIBEMITTER'] which is a list.
>>>>> >>>>>>>>> So if you append to SHLIBEMITTER and mark the relevant nodes
>>>>> >>>>>>>>> Precious()
>>>>> >>>>>>>>> env['PROGEMITTER'] is the emitter for programs. Looks like
>>>>> it's
>>>>> >>>>>>>>> only defined for mslink and qt, so you can defined one on
>>>>> non qt/mslink
>>>>> >>>>>>>>> freely and do the same.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Hope that helps
>>>>> >>>>>>>>> -Bill
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> On Fri, Apr 29, 2016 at 8:52 AM, William Blevins
>>>>> >>>>>>>>> <wblevins001 at gmail.com> wrote:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Andrew,
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> I expect that you could make a Pseudo-Builder that wraps
>>>>> Program
>>>>> >>>>>>>>>> and Library calls, and replace the top-level calls with
>>>>> your wrappers.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> I think you can replace them like env['BUILDERS']['xxxxxx']
>>>>> =
>>>>> >>>>>>>>>> Wrapper or env.AddMethod but I haven't tested this
>>>>> explicitly.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> https://bitbucket.org/scons/scons/wiki/ToolsForFools
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> V/R,
>>>>> >>>>>>>>>> William
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> On Fri, Apr 29, 2016 at 4:21 PM, Andrew C. Morrow
>>>>> >>>>>>>>>> <andrew.c.morrow at gmail.com> wrote:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Hi -
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Is there a simple way to do this? I'd like to use ld.gold's
>>>>> >>>>>>>>>>> incremental linking, but I don't want to need to add
>>>>> .Precious to every
>>>>> >>>>>>>>>>> executable and library in my project.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Thanks,
>>>>> >>>>>>>>>>> Andrew
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> _______________________________________________
>>>>> >>>>>>>>>>> Scons-users mailing list
>>>>> >>>>>>>>>>> Scons-users at scons.org
>>>>> >>>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> _______________________________________________
>>>>> >>>>>>>>>> Scons-users mailing list
>>>>> >>>>>>>>>> Scons-users at scons.org
>>>>> >>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> _______________________________________________
>>>>> >>>>>>>>> Scons-users mailing list
>>>>> >>>>>>>>> Scons-users at scons.org
>>>>> >>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >>>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> _______________________________________________
>>>>> >>>>>>>> Scons-users mailing list
>>>>> >>>>>>>> Scons-users at scons.org
>>>>> >>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >>>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> _______________________________________________
>>>>> >>>>>>> Scons-users mailing list
>>>>> >>>>>>> Scons-users at scons.org
>>>>> >>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >>>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> _______________________________________________
>>>>> >>>>>> Scons-users mailing list
>>>>> >>>>>> Scons-users at scons.org
>>>>> >>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >>>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> _______________________________________________
>>>>> >>>>> Scons-users mailing list
>>>>> >>>>> Scons-users at scons.org
>>>>> >>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>> _______________________________________________
>>>>> >>> Scons-users mailing list
>>>>> >>> Scons-users at scons.org
>>>>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> Scons-users mailing list
>>>>> >> Scons-users at scons.org
>>>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >>
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > Scons-users mailing list
>>>>> > Scons-users at scons.org
>>>>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> "I'm working on it."
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160517/68831f44/attachment-0001.html>