[Scons-users] Possible threading bug in scons

[Bill Deegan]

Tom,

Are the missing files the script depends on generated by the build?
So you need the output from this script to determine the files it depends
on?
So is this depending on some files written by swig or something like that?

Perhaps if you can provide more details an alternative to running Command()
in the scanner can be found..
-Bill

On Tue, Jan 19, 2016 at 1:03 PM, Tom Tanner (BLOOMBERG/ LONDON) <
ttanner2 at bloomberg.net> wrote:

> I think it's some sort of problem in the dependency scanning. There's a
> file (python), it then calls a dependency scanner but some of the files the
> script depends on don't yet exist so he wants the dependency scanning to be
> postponed till the files do exist.
>
> (Dependency scanning in python terms being - what python modules do you
> depend on, which we use for running pylint and unit tests automagically,
> but most of our builds don't actually depend on generating the files in the
> first place).
>
> From: bill at baddogconsulting.com At: Jan 19 2016 13:45:51
> To: Tom Tanner (BLOOMBERG/ LONDON) <ttanner2 at bloomberg.net>,
> scons-users at scons.org
> Cc: wblevins001 at gmail.com
>
> Subject: Re: [Scons-users] Possible threading bug in scons
>
> Tom,
>
> I'd say it's not advised.
> What are they trying to do in the Command in a scanner?
>
> -Bill
>
> On Tue, Jan 19, 2016 at 5:34 AM, Tom Tanner (BLOOMBERG/ LONDON) <
> ttanner2 at bloomberg.net> wrote:
>
>> We've narrowed it down to some code he changed in a scanner which appears
>> to be now adding an env.Command()
>>
>> Is that supported?
>>
>> Traceback (most recent call last):
>> File "scons/lib/scons-2.2.0/SCons/Taskmaster.py", line 233, in execute
>> if not t.retrieve_from_cache():
>> File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 2744, in
>> retrieve_from_cache
>> return self.get_build_env().get_CacheDir().retrieve(self)
>> File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 208, in retrieve
>> if CacheRetrieve(node, [], env, execute=1) == 0:
>> File "scons/lib/scons-2.2.0/SCons/Action.py", line 566, in __call__
>> cmd = self.strfunction(target, source, env)
>> File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 65, in
>> CacheRetrieveString
>> cachedir, cachefile = cd.cachepath(t)
>> File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 169, in cachepath
>> sig = node.get_cachedir_bsig()
>> File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 3127, in
>> get_cachedir_bsig
>> sigs = [n.get_cachedir_csig() for n in children]
>> File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 3107, in
>> get_cachedir_csig
>> cachedir, cachefile = self.get_build_env().get_CacheDir().cachepath(self)
>> File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 169, in cachepath
>> sig = node.get_cachedir_bsig()
>> File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 3127, in
>> get_cachedir_bsig
>> sigs = [n.get_cachedir_csig() for n in children]
>> File "scons/lib/scons-2.2.0/SCons/Node/__init__.py", line 787, in
>> get_cachedir_csig
>> return self.get_csig()
>> File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 1778, in get_csig
>> contents = self.get_contents()
>> File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 1769, in get_contents
>> contents.append('%s %s\n' % (node.get_csig(), node.name))
>> File "scons/lib/scons-2.2.0/SCons/Node/__init__.py", line 784, in
>> get_csig
>> return self.ninfo.csig
>> AttributeError: 'FileNodeInfo' object has no attribute 'csig'
>>
>> From: wblevins001 at gmail.com At: Jan 18 2016 22:20:44
>> To: Tom Tanner (BLOOMBERG/ LONDON) <ttanner2 at bloomberg.net>,
>> scons-users at scons.org
>> Subject: Re: [Scons-users] Possible threading bug in scons
>>
>> I looked through TaskMaster::execute in the latest and didn't see
>> anything that jumped out at me. Do you have a full stack trace or just that
>> snippet?
>>
>> On Mon, Jan 18, 2016 at 9:51 PM, William Blevins <wblevins001 at gmail.com>
>> wrote:
>>
>>> I'll take a brief look, but I am pretty busy during exam season. I have
>>> ran heterogeneous SCons builds on 40-core servers without every running
>>> into this, but it is still possible...
>>>
>>> On Mon, Jan 18, 2016 at 9:46 PM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Jan 18, 2016 at 2:48 PM, Tom Tanner (BLOOMBERG/ LONDON) <
>>>> ttanner2 at bloomberg.net> wrote:
>>>>
>>>>> Someone here has been doing a lot of parallel builds and come across
>>>>> what looks like a bug though I'm not sure why:
>>>>>
>>>>> The problem is with this bit in Taskmaster:
>>>>>
>>>>> def execute(self):
>>>>> """
>>>>> Called to execute the task.
>>>>>
>>>>> This method is called from multiple threads in a parallel build,
>>>>> so only do thread safe stuff here. Do thread unsafe stuff in
>>>>> prepare(), executed() or failed().
>>>>> """
>>>>> T = self.tm.trace
>>>>> if T: T.write(self.trace_message(u'Task.execute()', self.node))
>>>>>
>>>>> try:
>>>>> cached_targets = []
>>>>> for t in self.targets:
>>>>> if not t.retrieve_from_cache():
>>>>> break
>>>>>
>>>>> The t.retrieve_from_cache ends up doing this:
>>>>>
>>>>> File ".../SCons/Node/FS.py", line 1778, in get_csig
>>>>> contents = self.get_contents()
>>>>> File ".../SCons/Node/FS.py", line 1769, in get_contents
>>>>> contents.append('%s %s\n' % (node.get_csig(), node.name))
>>>>> File ".../SCons/Node/__init__.py", line 784, in get_csig
>>>>> return self.ninfo.csig
>>>>> AttributeError: 'FileNodeInfo' object has no attribute 'csig'
>>>>>
>>>>> which goes away if he puts a lock in Node.get_ninfo.
>>>>>
>>>>> Now, the only way I can think of this triggering is if he's managed to
>>>>> get two rules with the same target. Is that possible? and if so how can you
>>>>> detect it? I don't like the idea of having a lock in Node.get_ninfo.
>>>>>
>>>>
>>>> Two rules with the same target generate a SCons Error at SConscript
>>>> parser time.
>>>>
>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160119/94b05fd5/attachment-0001.html>