[Scons-users] Possible threading bug in scons

Tue Jan 19 13:03:39 EST 2016

I think it's some sort of problem in the dependency scanning. There's a file (python), it then calls a dependency scanner but some of the files the script depends on don't yet exist so he wants the dependency scanning to be postponed till the files do exist.

(Dependency scanning in python terms being - what python modules do you depend on, which we use for running pylint and unit tests automagically, but most of our builds don't actually depend on generating the files in the first place).

From: bill at baddogconsulting.com At: Jan 19 2016 13:45:51
To: Tom Tanner (BLOOMBERG/ LONDON), scons-users at scons.org
Cc: wblevins001 at gmail.com
Subject: Re: [Scons-users] Possible threading bug in scons

Tom,

I'd say it's not advised.
What are they trying to do in the Command in a scanner?

-Bill

On Tue, Jan 19, 2016 at 5:34 AM, Tom Tanner (BLOOMBERG/ LONDON) <ttanner2 at bloomberg.net> wrote:

We've narrowed it down to some code he changed in a scanner which appears to be now adding an env.Command()

Is that supported?

Traceback (most recent call last):
  File "scons/lib/scons-2.2.0/SCons/Taskmaster.py", line 233, in execute
    if not t.retrieve_from_cache():
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 2744, in retrieve_from_cache
    return self.get_build_env().get_CacheDir().retrieve(self)
  File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 208, in retrieve
    if CacheRetrieve(node, [], env, execute=1) == 0:
  File "scons/lib/scons-2.2.0/SCons/Action.py", line 566, in __call__
    cmd = self.strfunction(target, source, env)
  File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 65, in CacheRetrieveString
    cachedir, cachefile = cd.cachepath(t)
  File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 169, in cachepath
    sig = node.get_cachedir_bsig()
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 3127, in get_cachedir_bsig
    sigs = [n.get_cachedir_csig() for n in children]
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 3107, in get_cachedir_csig
    cachedir, cachefile = self.get_build_env().get_CacheDir().cachepath(self)
  File "scons/lib/scons-2.2.0/SCons/CacheDir.py", line 169, in cachepath
    sig = node.get_cachedir_bsig()
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 3127, in get_cachedir_bsig
    sigs = [n.get_cachedir_csig() for n in children]
  File "scons/lib/scons-2.2.0/SCons/Node/__init__.py", line 787, in get_cachedir_csig
    return self.get_csig()
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 1778, in get_csig
    contents = self.get_contents()
  File "scons/lib/scons-2.2.0/SCons/Node/FS.py", line 1769, in get_contents
    contents.append('%s %s\n' % (node.get_csig(), node.name))
  File "scons/lib/scons-2.2.0/SCons/Node/__init__.py", line 784, in get_csig
    return self.ninfo.csig
AttributeError: 'FileNodeInfo' object has no attribute 'csig'

From: wblevins001 at gmail.com At: Jan 18 2016 22:20:44To: Tom Tanner (BLOOMBERG/ LONDON), scons-users at scons.org
Subject: Re: [Scons-users] Possible threading bug in scons

I looked through TaskMaster::execute in the latest and didn't see anything that jumped out at me. Do you have a full stack trace or just that snippet?

On Mon, Jan 18, 2016 at 9:51 PM, William Blevins <wblevins001 at gmail.com> wrote:

I'll take a brief look, but I am pretty busy during exam season. I have ran heterogeneous SCons builds on 40-core servers without every running into this, but it is still possible...

On Mon, Jan 18, 2016 at 9:46 PM, William Blevins <wblevins001 at gmail.com> wrote:

On Mon, Jan 18, 2016 at 2:48 PM, Tom Tanner (BLOOMBERG/ LONDON) <ttanner2 at bloomberg.net> wrote:

Someone here has been doing a lot of parallel builds and come across what looks like a bug though I'm not sure why:

The problem is with this bit in Taskmaster:

    def execute(self):
        """
        Called to execute the task.

        This method is called from multiple threads in a parallel build,
        so only do thread safe stuff here.  Do thread unsafe stuff in
        prepare(), executed() or failed().
        """
        T = self.tm.trace
        if T: T.write(self.trace_message(u'Task.execute()', self.node))

        try:
            cached_targets = []
            for t in self.targets:
                if not t.retrieve_from_cache():
                    break

The t.retrieve_from_cache ends up doing this:

  File ".../SCons/Node/FS.py", line 1778, in get_csig
    contents = self.get_contents()
  File ".../SCons/Node/FS.py", line 1769, in get_contents
    contents.append('%s %s\n' % (node.get_csig(), node.name))
  File ".../SCons/Node/__init__.py", line 784, in get_csig
    return self.ninfo.csig
AttributeError: 'FileNodeInfo' object has no attribute 'csig'

which goes away if he puts a lock in Node.get_ninfo.

Now, the only way I can think of this triggering is if he's managed to get two rules with the same target. Is that possible? and if so how can you detect it? I don't like the idea of having a lock in Node.get_ninfo.

Two rules with the same target generate a SCons Error at SConscript parser time.

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160119/412eb119/attachment.html>