[Scons-users] Possible threading bug in scons

[Tom Tanner (BLOOMBERG/ LONDON)]

We've narrowed it down to some code he changed in a scanner which appears to be now adding an env.Command()

Is that supported?

Traceback (most recent call last):
  File "scons/lib/scons-2.2.0/SCons/Taskmaster.py", line 233, in execute
    if not t.retrieve_from_cache():
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 2744, in retrieve_from_cache
    return self.get_build_env().get_CacheDir().retrieve(self)
  File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 208, in retrieve
    if CacheRetrieve(node, [], env, execute=1) == 0:
  File "scons/lib/scons-2.2.0/SCons/Action.py", line 566, in __call__
    cmd = self.strfunction(target, source, env)
  File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 65, in CacheRetrieveString
    cachedir, cachefile = cd.cachepath(t)
  File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 169, in cachepath
    sig = node.get_cachedir_bsig()
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 3127, in get_cachedir_bsig
    sigs = [n.get_cachedir_csig() for n in children]
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 3107, in get_cachedir_csig
    cachedir, cachefile = self.get_build_env().get_CacheDir().cachepath(self)
  File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 169, in cachepath
    sig = node.get_cachedir_bsig()
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 3127, in get_cachedir_bsig
    sigs = [n.get_cachedir_csig() for n in children]
  File "scons/lib/scons-2.2.0/SCons/Node/__init__.py", line 787, in get_cachedir_csig
    return self.get_csig()
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 1778, in get_csig
    contents = self.get_contents()
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 1769, in get_contents
    contents.append('%s %s\n' % (node.get_csig(), node.name))
  File "scons/lib/scons-2.2.0/SCons/Node/__init__.py", line 784, in get_csig
    return self.ninfo.csig
AttributeError: 'FileNodeInfo' object has no attribute 'csig'

From: wblevins001 at gmail.com At: Jan 18 2016 22:20:44
To: Tom Tanner (BLOOMBERG/ LONDON), scons-users at scons.org
Subject: Re: [Scons-users] Possible threading bug in scons

I looked through TaskMaster::execute in the latest and didn't see anything that jumped out at me. Do you have a full stack trace or just that snippet?

On Mon, Jan 18, 2016 at 9:51 PM, William Blevins <wblevins001 at gmail.com> wrote:

I'll take a brief look, but I am pretty busy during exam season. I have ran heterogeneous SCons builds on 40-core servers without every running into this, but it is still possible...

On Mon, Jan 18, 2016 at 9:46 PM, William Blevins <wblevins001 at gmail.com> wrote:


On Mon, Jan 18, 2016 at 2:48 PM, Tom Tanner (BLOOMBERG/ LONDON) <ttanner2 at bloomberg.net> wrote:

Someone here has been doing a lot of parallel builds and come across what looks like a bug though I'm not sure why:

The problem is with this bit in Taskmaster:

    def execute(self):
        """
        Called to execute the task.

        This method is called from multiple threads in a parallel build,
        so only do thread safe stuff here.  Do thread unsafe stuff in
        prepare(), executed() or failed().
        """
        T = self.tm.trace
        if T: T.write(self.trace_message(u'Task.execute()', self.node))

        try:
            cached_targets = []
            for t in self.targets:
                if not t.retrieve_from_cache():
                    break

The t.retrieve_from_cache ends up doing this:

  File ".../SCons/Node/FS.py", line 1778, in get_csig
    contents = self.get_contents()
  File ".../SCons/Node/FS.py", line 1769, in get_contents
    contents.append('%s %s\n' % (node.get_csig(), node.name))
  File ".../SCons/Node/__init__.py", line 784, in get_csig
    return self.ninfo.csig
AttributeError: 'FileNodeInfo' object has no attribute 'csig'

which goes away if he puts a lock in Node.get_ninfo.

Now, the only way I can think of this triggering is if he's managed to get two rules with the same target. Is that possible? and if so how can you detect it? I don't like the idea of having a lock in Node.get_ninfo.

Two rules with the same target generate a SCons Error at SConscript parser time.
 

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160119/a2cb4a38/attachment.html>