[Scons-users] Make all Program and SharedLibrary targets implicitly Precious?
Andrew C. Morrow
andrew.c.morrow at gmail.com
Tue May 17 16:27:00 EDT 2016
Hi -
Just to clarify - was it just -Wl,-fuse-ld=gold that you tried and got the
speedup, or the whole incremental linking tool? I personally wasn't able to
get incremental binaries built with gold to work at all - they just crashed
in early startup. I probably should file some binutils tickets...
Thanks,
Andrew
On Tue, May 17, 2016 at 4:47 AM, Ale Strooisma <
a.strooisma at student.utwente.nl> wrote:
> Hi Andrew,
>
> I actually needed to do LINKFLAGS='-Wl,-fuse-ld=gold' and I must say it
> gives an insane speed-up. I am really happy I found out about this tool!
>
> Kind regards,
> Ale Strooisma
>
> On 13 May 2016 at 15:47, Andrew C. Morrow <andrew.c.morrow at gmail.com>
> wrote:
>
>>
>> Hi -
>>
>> The equivalent of env.Append(LINKFLAGS=['-fuse-ld=gold']) is what MongoDB
>> uses to enable the gold linker:
>>
>>
>> https://github.com/mongodb/mongo/blob/d9a8e6a9db1bd48c2bbfb1ad18e7cb8e18eb7302/SConstruct#L2164-L2167
>>
>> You almost certainly don't want to change LINK in a C++ build, for the
>> reasons you found. SCons is really fairly smart about how to invoke the
>> linker correctly (search for SmartLink).
>>
>> Hope this helps,
>> Andrew
>>
>>
>> On Fri, May 13, 2016 at 4:56 AM, Ale Strooisma <
>> a.strooisma at student.utwente.nl> wrote:
>>
>>> Ah okay, that works, thanks! Silly that I did not come up with that
>>> myself.
>>> Still a few more libraries to add, though. I'm definitely going to try
>>> your alternative.
>>>
>>> Kind regards,
>>> Ale Strooisma
>>>
>>> On 13 May 2016 at 10:44, Jörg Plate <patterner at gmail.com> wrote:
>>>
>>>> The stdc++ library tends to be in non-standard locations, since it's
>>>> depending on the compiler.
>>>> Try
>>>> locate "*libstdc++*so*"
>>>> and use the -L option with the correct directory. My g++ uses
>>>> /usr/lib/gcc/x86_64-linux-gnu/6/.
>>>>
>>>> Alternative: use g++ with "-Wl,-fuse-ld=gold" and let g++ deal with
>>>> it.
>>>>
>>>> On Fri, May 13, 2016 at 10:30 AM, Ale Strooisma
>>>> <a.strooisma at student.utwente.nl> wrote:
>>>> > Andrew,
>>>> >
>>>> > inspired by this discussion I decided to try using gold as well. I
>>>> ran into
>>>> > a problem however, and can't seem to find a lot of documentation on
>>>> gold, so
>>>> > I hoped you could shed some light on this problem as you are working
>>>> with
>>>> > gold and scons.
>>>> >
>>>> > I changed to gold by adding LINK='ld.gold' to my environment object.
>>>> > However, now I get a lot of undefined reference errors during the
>>>> linking
>>>> > process - they all seem to come from the standard libraries. I
>>>> interpret
>>>> > this as gold not linking those by default, so I added stdc++ to LIBS,
>>>> but
>>>> > that resulted in "ld.gold: error: cannot find -lstdc++".
>>>> >
>>>> > Any tips for getting this to work?
>>>> >
>>>> > Kind regards,
>>>> > Ale Strooisma
>>>> >
>>>> >
>>>> > On 13 May 2016 at 00:28, Bill Deegan <bill at baddogconsulting.com>
>>>> wrote:
>>>> >>
>>>> >> Andrew,
>>>> >>
>>>> >> Looks reasonable.
>>>> >> What happens to the old binary if you os.remove it on one system,
>>>> when
>>>> >> it's being used on another? (NFS)
>>>> >> Does it crash, or is the file kept alive via inode reference by
>>>> running
>>>> >> process?
>>>> >>
>>>> >> -Bill
>>>> >>
>>>> >> On Thu, May 12, 2016 at 4:08 PM, Andrew C. Morrow
>>>> >> <andrew.c.morrow at gmail.com> wrote:
>>>> >>>
>>>> >>>
>>>> >>> OK, I got something that sort of works for incremental linking. I'm
>>>> sure
>>>> >>> it can be improved:
>>>> >>>
>>>> >>>
>>>> >>>
>>>> https://github.com/acmorrow/mongo/blob/0062b8bfa0c24812438f4765fa20755fb5f6fc54/site_scons/site_tools/incremental_link.py
>>>> >>>
>>>> >>> The binaries that ld.gold produces seem terribly broken, but
>>>> perhaps it
>>>> >>> will be useful on Windows.
>>>> >>>
>>>> >>> I wonder if .Precious should have a mode where it behaves like this
>>>> by
>>>> >>> default, or if it should take an optional parameter. It seems
>>>> useful.
>>>> >>>
>>>> >>> On Thu, May 12, 2016 at 12:18 PM, Andrew C. Morrow
>>>> >>> <andrew.c.morrow at gmail.com> wrote:
>>>> >>>>
>>>> >>>>
>>>> >>>> Hi Bill -
>>>> >>>>
>>>> >>>> I like this suggestion of adding a PreAction to each executable and
>>>> >>>> shared library, but I don't want to have to explicitly call
>>>> .AddPreAction on
>>>> >>>> each target.
>>>> >>>>
>>>> >>>> Can you suggest an easy way to achieve the same effect globally
>>>> for all
>>>> >>>> SharedLibrary and Program targets, without writing a
>>>> PseudoBuilder? I'm fine
>>>> >>>> though with reaching into the existing builders and altering them.
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>> Andrew
>>>> >>>>
>>>> >>>> On Sat, Apr 30, 2016 at 4:38 PM, Bill Deegan <
>>>> bill at baddogconsulting.com>
>>>> >>>> wrote:
>>>> >>>>>
>>>> >>>>> Andrew,
>>>> >>>>>
>>>> >>>>> How about a pre-action to mv the existing file to another name,
>>>> and
>>>> >>>>> then copy it to original name, then incremental link the copy?
>>>> >>>>> That should resolve the crashing bit.
>>>> >>>>>
>>>> >>>>> lsof filename won't help if the files being used on another
>>>> system, but
>>>> >>>>> would if on current system. (to see if in use)
>>>> >>>>>
>>>> >>>>> -Bill
>>>> >>>>>
>>>> >>>>> On Sat, Apr 30, 2016 at 3:47 PM, Andrew C. Morrow
>>>> >>>>> <andrew.c.morrow at gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> Definitely a possibility. MongoDB probes for compiler support for
>>>> >>>>>> ld.gold and prefers it by default:
>>>> >>>>>>
>>>> >>>>>>
>>>> https://github.com/mongodb/mongo/blob/master/SConstruct#L2164-L2167
>>>> >>>>>>
>>>> >>>>>> The downside to using incremental linking and making dynamic
>>>> targets
>>>> >>>>>> Precious is that by overwriting the binaries you end up crashing
>>>> running
>>>> >>>>>> programs that have them mapped. That happens often enough during
>>>> developer
>>>> >>>>>> workflow that it is unfortunate.
>>>> >>>>>>
>>>> >>>>>> So, it definitely should be opt-in behavior, and a Tool sounds
>>>> right
>>>> >>>>>> for that.
>>>> >>>>>>
>>>> >>>>>> I also wonder whether such a Tool might be able to make
>>>> intelligent
>>>> >>>>>> use of gold's --incremental-base file to avoid the above
>>>> problem. If you
>>>> >>>>>> first copied the dynamic target aside under a different name,
>>>> then unlinked
>>>> >>>>>> the original, then used --incremental-base pointing to the
>>>> moved-aside file,
>>>> >>>>>> perhaps you could avoid crashing programs that have the target
>>>> mapped. I'm
>>>> >>>>>> not sure though if the cost of the file copy would eat up most
>>>> of the gains
>>>> >>>>>> from incremental linking.
>>>> >>>>>>
>>>> >>>>>> That sounds complex enough to also warrant a separate Tool.
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Fri, Apr 29, 2016 at 1:52 PM, Bill Deegan
>>>> >>>>>> <bill at baddogconsulting.com> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Andrew,
>>>> >>>>>>>
>>>> >>>>>>> Might make sense to make a gold linker tool?
>>>> >>>>>>>
>>>> >>>>>>> -Bill
>>>> >>>>>>>
>>>> >>>>>>> On Fri, Apr 29, 2016 at 9:47 AM, Andrew C. Morrow
>>>> >>>>>>> <andrew.c.morrow at gmail.com> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> Thanks Bill -
>>>> >>>>>>>>
>>>> >>>>>>>> I think that sounds promising, I'll give it a look. If I get it
>>>> >>>>>>>> working well I'll ping the list back and describe my findings.
>>>> >>>>>>>>
>>>> >>>>>>>> Thanks,
>>>> >>>>>>>> Andrew
>>>> >>>>>>>>
>>>> >>>>>>>> On Fri, Apr 29, 2016 at 12:10 PM, Bill Deegan
>>>> >>>>>>>> <bill at baddogconsulting.com> wrote:
>>>> >>>>>>>>>
>>>> >>>>>>>>> Andrew,
>>>> >>>>>>>>>
>>>> >>>>>>>>> Likely you want to stick this logic in the emitter for
>>>> building
>>>> >>>>>>>>> libraries and programs.
>>>> >>>>>>>>> SCons/Tool/link.py shlib_emitter() is the current logic.
>>>> >>>>>>>>> The environment tracks this in:
>>>> >>>>>>>>> env['SHLIBEMITTER'] which is a list.
>>>> >>>>>>>>> So if you append to SHLIBEMITTER and mark the relevant nodes
>>>> >>>>>>>>> Precious()
>>>> >>>>>>>>> env['PROGEMITTER'] is the emitter for programs. Looks like
>>>> it's
>>>> >>>>>>>>> only defined for mslink and qt, so you can defined one on non
>>>> qt/mslink
>>>> >>>>>>>>> freely and do the same.
>>>> >>>>>>>>>
>>>> >>>>>>>>> Hope that helps
>>>> >>>>>>>>> -Bill
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Fri, Apr 29, 2016 at 8:52 AM, William Blevins
>>>> >>>>>>>>> <wblevins001 at gmail.com> wrote:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Andrew,
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> I expect that you could make a Pseudo-Builder that wraps
>>>> Program
>>>> >>>>>>>>>> and Library calls, and replace the top-level calls with your
>>>> wrappers.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> I think you can replace them like env['BUILDERS']['xxxxxx'] =
>>>> >>>>>>>>>> Wrapper or env.AddMethod but I haven't tested this
>>>> explicitly.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> https://bitbucket.org/scons/scons/wiki/ToolsForFools
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> V/R,
>>>> >>>>>>>>>> William
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On Fri, Apr 29, 2016 at 4:21 PM, Andrew C. Morrow
>>>> >>>>>>>>>> <andrew.c.morrow at gmail.com> wrote:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Hi -
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Is there a simple way to do this? I'd like to use ld.gold's
>>>> >>>>>>>>>>> incremental linking, but I don't want to need to add
>>>> .Precious to every
>>>> >>>>>>>>>>> executable and library in my project.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Thanks,
>>>> >>>>>>>>>>> Andrew
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> _______________________________________________
>>>> >>>>>>>>>>> Scons-users mailing list
>>>> >>>>>>>>>>> Scons-users at scons.org
>>>> >>>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> _______________________________________________
>>>> >>>>>>>>>> Scons-users mailing list
>>>> >>>>>>>>>> Scons-users at scons.org
>>>> >>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >>>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>> _______________________________________________
>>>> >>>>>>>>> Scons-users mailing list
>>>> >>>>>>>>> Scons-users at scons.org
>>>> >>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> _______________________________________________
>>>> >>>>>>>> Scons-users mailing list
>>>> >>>>>>>> Scons-users at scons.org
>>>> >>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >>>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> _______________________________________________
>>>> >>>>>>> Scons-users mailing list
>>>> >>>>>>> Scons-users at scons.org
>>>> >>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> _______________________________________________
>>>> >>>>>> Scons-users mailing list
>>>> >>>>>> Scons-users at scons.org
>>>> >>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> _______________________________________________
>>>> >>>>> Scons-users mailing list
>>>> >>>>> Scons-users at scons.org
>>>> >>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >>>>>
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>> _______________________________________________
>>>> >>> Scons-users mailing list
>>>> >>> Scons-users at scons.org
>>>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >>>
>>>> >>
>>>> >>
>>>> >> _______________________________________________
>>>> >> Scons-users mailing list
>>>> >> Scons-users at scons.org
>>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >>
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Scons-users mailing list
>>>> > Scons-users at scons.org
>>>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> "I'm working on it."
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160517/12d81cbb/attachment-0001.html>
More information about the Scons-users
mailing list