[Scons-users] CudaTool

Thu Jun 18 23:10:37 EDT 2015

All,

The CUDA Tool https://bitbucket.org/scons/scons/wiki/CudaTool has been
updated to resolve the previously annoying linkinfo file handling :)

V/R,
William

On Thu, Jun 18, 2015 at 10:49 PM, William Blevins <wblevins001 at gmail.com>
wrote:

> David,
>
> I assume you checked that the linkinfo showed as dependencies or their
> respective ".cu" source file and they were correctly cleaned with "scons
> -c" ;)
>
> V/R,
> William
>
> On Thu, Jun 18, 2015 at 10:43 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
>
>> David,
>>
>> No worries. I am simply glad that you responded.  I have a huge stack of
>> working items, and most of them depend on user feedback, so getting some
>> issues finished is always rewarding~!
>>
>> I will update the wiki immediately.
>>
>> Just I guy trying to make a difference,
>> William
>>
>>
>>
>> On Thu, Jun 18, 2015 at 11:47 AM, David Wade <DAWAD at statoil.com> wrote:
>>
>>>  Hi William,
>>>
>>>
>>>
>>> There were some other priorities earlier today while your original
>>> workaround was holding things together okay…
>>>
>>> I’ve now found a moment to look at this, and it works like a charm.
>>> SConscript back to being simple and intuitive – thank you!
>>>
>>> I would say the new cuda.py is ready to go up on you wiki for everyone
>>> else J
>>>
>>>
>>>
>>> Best regards,
>>>
>>> David
>>>
>>>
>>>
>>> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf
>>> Of *William Blevins
>>> *Sent:* 17 June 2015 19:01
>>>
>>> *To:* SCons users mailing list
>>> *Subject:* Re: [Scons-users] CudaTool
>>>
>>>
>>>
>>> David,
>>>
>>>
>>>
>>> So rereading your email.  After the env fix, the linkinfo isn't giving
>>> you any issues or are you still having to workaround it?
>>>
>>>
>>>
>>> V/R,
>>>
>>> William
>>>
>>>
>>>
>>> On Wed, Jun 17, 2015 at 11:41 AM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>> David and Bill,
>>>
>>>
>>>
>>> After Bill and I discussed the linkinfo files for a while and I did some
>>> googling, I came to the conclusion that the linkinfo files shouldn't be
>>> returned from the emitters because I don't think they are even used:
>>> https://devtalk.nvidia.com/default/topic/405518/linkinfo-files
>>>
>>>
>>>
>>> def CUDANVCCStaticObjectEmitter(target, source, env):
>>>         tgt, src = SCons.Defaults.StaticObjectEmitter(target, source,
>>> env)
>>>         for file in src:
>>>                 lifile = os.path.splitext(src[0].rstr())[0] + '.linkinfo'
>>>                 tgt.append(lifile)
>>>         return tgt, src
>>> def CUDANVCCSharedObjectEmitter(target, source, env):
>>>         tgt, src = SCons.Defaults.SharedObjectEmitter(target, source,
>>> env)
>>>         for file in src:
>>>                 lifile = os.path.splitext(src[0].rstr())[0] + '.linkinfo'
>>>                 tgt.append(lifile)
>>>         return tgt, src
>>>
>>>
>>>
>>> For these reasons, I believe we can tweak the Emitters to handle
>>> cleaning and dependencies correctly without returning the linkinfo files in
>>> the emitters meaning you won't need to muck with the return values from the
>>> builders.  This will help keep your SConscripts simple which is always a
>>> good thing.
>>>
>>>
>>>
>>> David,
>>>
>>>
>>>
>>> Can you try the following?  I don't have a setup for CUDA, but I believe
>>> this will be a good improvement over the current wiki implementation.
>>> Update the emitters like so:
>>>
>>>
>>>
>>> def CUDANVCCStaticObjectEmitter(target, source, env):
>>>         tgt, src = SCons.Defaults.StaticObjectEmitter(target, source,
>>> env)
>>>         for file in tgt:
>>>                 lifile = os.path.splitext(file.rstr())[0] + '.linkinfo'
>>>                 env.SideEffect( lifile, file )
>>>                 env.Clean( file, lifile )
>>>
>>>          return tgt, src
>>>
>>>
>>>
>>>  def CUDANVCCSharedObjectEmitter(target, source, env):
>>>         tgt, src = SCons.Defaults.SharedObjectEmitter(target, source,
>>> env)
>>>         for file in tgt:
>>>                 lifile = os.path.splitext(file.rstr())[0] + '.linkinfo'
>>>                 env.SideEffect( lifile, file )
>>>                 env.Clean( file, lifile )
>>>         return tgt, src
>>>
>>>
>>>
>>> Bill,
>>>
>>>
>>>
>>> Can you double check my patch to make sure I'm not doing anything silly
>>> like have the Clean/SideEffect parameters in the wrong order?  Also, I am
>>> pretty sure the previous loop logic didn't behave correctly for multiple
>>> sources and/or if the targets weren't put into the same directory...
>>>
>>>
>>>
>>> Just some guy wanting to make technology fun,
>>>
>>> William
>>>
>>>
>>>
>>> On Wed, Jun 17, 2015 at 2:41 AM, David Wade <DAWAD at statoil.com> wrote:
>>>
>>> … and from the land of CET, good morning!
>>>
>>>
>>>
>>> Thanks to both of you for looking into this for me. I would indeed have
>>> gone to the developer of the SCons CUDA tool first, but unfortunately I
>>> couldn’t find who that is! I think if you’re able to figure out who wrote
>>> it and ask them to investigate when/if the .linkinfo file is needed in the
>>> toolchain that would be great.
>>>
>>>
>>>
>>> However, using your suggestions for a better workaround (you’re quite
>>> right I made a typo with env1 twice) I’ve been able to revert to the
>>> original cuda.py (although for reference I used str(x) rather than x.str()
>>> ).
>>>
>>>
>>>
>>> And… as a bonus I now have a better idea what SCons is doing in there!
>>>
>>>
>>>
>>> Thanks again!
>>>
>>> David
>>>
>>>
>>>
>>> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf
>>> Of *William Blevins
>>> *Sent:* 17 June 2015 01:18
>>> *To:* SCons users mailing list
>>> *Subject:* Re: [Scons-users] CudaTool
>>>
>>>
>>>
>>> Bill,
>>>
>>>
>>>
>>> The SideEffect documentation states that sideeffects are not cleaned,
>>> and Clean must be called explicitly, so maybe we need to do both.
>>>
>>>
>>>
>>> V/R,
>>>
>>> William
>>>
>>>
>>>
>>> On Tue, Jun 16, 2015 at 5:32 PM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>> Bill,
>>>
>>>
>>>
>>> Hmm.  Unless I am missing something, the entirety of the Tool exists on
>>> the Wiki, so if it doesn't add it there then it isn't used.  I haven't
>>> heard of that file extension from any other compiler.
>>>
>>>
>>>
>>> David,
>>>
>>>
>>>
>>> Can you try making the change Bill and I are discussing.  See if that
>>> works for you?   If so, I imagine this will simplify your build scripts a
>>> lot.  We can make an update to the SCons wiki also.  Let me know if you
>>> need a delta explicitly.
>>>
>>>
>>>
>>> V/R,
>>>
>>> William
>>>
>>>
>>>
>>> On Tue, Jun 16, 2015 at 5:03 PM, Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>> Likely the right way for the Cuda tool to handle this is to mark those
>>> files as SideEffect(), unless they are somehow an input to other tools..
>>>
>>> Similar issues happen with env.SharedLibrary() on win32.. You'll get a
>>> .dll, .lib and/or .def file so if you want to use the returned nodes you
>>> may have to filter them for the one you want.
>>>
>>> -Bill
>>>
>>>
>>>
>>> On Tue, Jun 16, 2015 at 1:57 PM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>> Bill,
>>>
>>>
>>>
>>> Or is the SConsWiki the original source, so we are the developers?
>>>
>>>
>>>
>>> V/R,
>>>
>>> William
>>>
>>>
>>>
>>> On Tue, Jun 16, 2015 at 4:55 PM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>> David,
>>>
>>>
>>>
>>> All in all, it seems that may have been a question more appropriate for
>>> the SCons CUDA tool developer, but looking at
>>> https://bitbucket.org/scons/scons/wiki/CudaTool notes section, it
>>> appears that those files are only added to the Emitter so that scons -c
>>> works for those files.
>>>
>>>
>>>
>>> Bill,
>>>
>>>
>>>
>>> Can the CUDA emitter simply call Clean( 'item.linkinfo' ) in the emitter
>>> to explicitly without needing to return the item itself?  I assume this
>>> would work, and it would simply that Builder if no one uses the
>>> "*.linkinfo" file.  If you think this will, I will contact the developer.
>>>
>>>
>>>
>>> V/R,
>>>
>>> William
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jun 16, 2015 at 11:00 AM, Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>> William,
>>>
>>> Thanks for the assist, you are correct in your corrections.
>>>
>>> Good question about the purpose of the .linkinfo. (They are added in the
>>> cuda tool..)
>>>
>>> I did take a course on cuda programming a while back and I"m thinking
>>> these are used to link the native code to code which will run on the GPU
>>> hardware.
>>>
>>> So while they (I believe) aren't needed in the .a they are used by other
>>> tools in the tool chain.
>>>
>>> -Bill
>>>
>>>
>>>
>>> On Tue, Jun 16, 2015 at 7:56 AM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>>
>>>
>>> On Tue, Jun 16, 2015 at 10:29 AM, Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>> David,
>>>
>>> How about:
>>>
>>> foo = env.StaticObject(“foo.cu”,<foo-options>)
>>> bar = env.StaticObject(“bar.cu”,<bar-options>)
>>>
>>> foo_objects = [ x for x in foo if x.str().endswith('.o') ]
>>> bar_objects = [ x for x in bar if x.str().endswith('.o') ]
>>>
>>>
>>> env.StaticLibrary(libname, [foo, bar]
>>>
>>>
>>>
>>> I think the about line needs to be env.StaticLibrary(libname,
>>> [foo_objects, bar_objects] though right?
>>>
>>>
>>>
>>>  [Also note your example you added bar-options to env1 instead of env2]
>>> I didn't try the code above but that's the general way I'd go about it.
>>> Remember that all builders return list of nodes.
>>>
>>> -Bill
>>>
>>>
>>>
>>> Thanks Bill.
>>>
>>>
>>>
>>> I wasn't 100% sure I understood his question, but even though we have
>>> given him a workaround I think we have sidestepped a more important
>>> question.
>>>
>>>
>>>
>>> What are *.linkinfo used for, and if they aren't being used why are they
>>> being created?  Are linkinfo specific to certain toolchains?
>>>
>>>
>>>
>>> V/R,
>>>
>>> William
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jun 16, 2015 at 5:31 AM, David Wade <DAWAD at statoil.com> wrote:
>>>
>>>   Hi,
>>>
>>>
>>>
>>> I have been using this nifty bit of python for some time:
>>> https://bitbucket.org/scons/scons/wiki/CudaTool
>>>
>>> Just ran into an issue of finding scons was trying to run “ar rc foo.o
>>> foo.linkinfo bar.o bar.linkinfo” when I tried this:
>>>
>>>
>>>
>>> env1 = env.Clone()
>>>
>>> env2 = env.Clone()
>>>
>>>
>>>
>>> env1.Append(<foo-options>)
>>>
>>> foo = env1.StaticObject(“foo.cu”)
>>>
>>> env1.Append(<bar-options>)
>>>
>>> bar = env2.StaticObject(“bar.cu”)
>>>
>>>
>>>
>>> env.StaticLibrary(libname, [foo, bar])
>>>
>>>
>>>
>>> My workaround has been to comment out references to the “Emitter”
>>> functions in cuda.py … but is this the best thing to do?
>>>
>>>
>>>
>>> Many thanks!
>>>
>>>
>>> *David Wade*
>>> Senior Analyst, Seismic Imaging Development
>>> *ITV SUB RH / HPC Development Team, *Statoil ASA
>>>
>>> *Phone*: +47 97572157
>>> *Email*: *dawad at statoil.com <dawad at statoil.com>*
>>> *Visitor address*: Vassbotnen 23, Forus, Norway
>>>
>>>
>>> Incorporation number: NO 923 609 016 MVA
>>> *www.statoil.com <http://www.statoil.com/>*
>>> Please consider the environment before printing this e-mail.
>>>
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------------------
>>> The information contained in this message may be CONFIDENTIAL and is
>>> intended for the addressee only. Any unauthorised use, dissemination of
>>> the
>>> information or copying of this message is prohibited. If you are not the
>>> addressee, please notify the sender immediately by return e-mail and
>>> delete
>>> this message.
>>> Thank you
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------------------
>>> The information contained in this message may be CONFIDENTIAL and is
>>> intended for the addressee only. Any unauthorised use, dissemination of
>>> the
>>> information or copying of this message is prohibited. If you are not the
>>> addressee, please notify the sender immediately by return e-mail and
>>> delete
>>> this message.
>>> Thank you
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------------------
>>> The information contained in this message may be CONFIDENTIAL and is
>>> intended for the addressee only. Any unauthorised use, dissemination of
>>> the
>>> information or copying of this message is prohibited. If you are not the
>>> addressee, please notify the sender immediately by return e-mail and
>>> delete
>>> this message.
>>> Thank you
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150618/cdf648c7/attachment-0001.html>