[Scons-users] scons-local-2.5.1 build and clean shows random failure

Tue Jul 25 04:52:01 EDT 2017

Any one has any new ideas? Seems this is a quite wide spread issue,
scons seems are not handling the cyclic dependency reliable enough on
parallel build.

On Sat, Jul 22, 2017 at 7:49 PM, Hua Yanghao <huayanghao at gmail.com> wrote:
> Yes, I trigger the single build when it fails, and generated the file.
> Just in case I generated it again and attached here.
>
> On Sat, Jul 22, 2017 at 6:08 PM, Bill Deegan <bill at baddogconsulting.com> wrote:
>> Is the taskmaster trace file from when you saw a failure?
>>
>> On Fri, Jul 21, 2017 at 11:47 PM, Hua Yanghao <huayanghao at gmail.com> wrote:
>>>
>>> By the way the failure is random, below 4 consecutive runs failed
>>> twice passed twice:
>>> hua at grass:~/git/usw $ tools/scons -j8 -c config=linux64_full.py
>>> scons: Reading SConscript files ...
>>> scons: done reading SConscript files.
>>> scons: Cleaning targets ...
>>> scons: done cleaning targets.
>>> hua at grass:~/git/usw $ tools/scons -j8 -c config=linux64_full.py
>>> scons: Reading SConscript files ...
>>> scons: done reading SConscript files.
>>> scons: Cleaning targets ...
>>> scons: done cleaning targets.
>>>
>>> scons: *** Found dependency cycle(s):
>>>   Internal Error: no cycle found for node output/linux64_full
>>> (<SCons.Node.FS.Dir object at 0x55d10f3ee5a0>) in state pending
>>>
>>> File "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/Taskmaster.py",
>>> line 1043, in cleanup
>>> hua at grass:~/git/usw $ tools/scons -j8 -c config=linux64_full.py
>>> scons: Reading SConscript files ...
>>> scons: done reading SConscript files.
>>> scons: Cleaning targets ...
>>> scons: done cleaning targets.
>>>
>>> scons: *** Found dependency cycle(s):
>>>   Internal Error: no cycle found for node output/linux64_full
>>> (<SCons.Node.FS.Dir object at 0x55f9ed59ef20>) in state pending
>>>
>>> File "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/Taskmaster.py",
>>> line 1043, in cleanup
>>> hua at grass:~/git/usw $ tools/scons -j8 -c config=linux64_full.py
>>> scons: Reading SConscript files ...
>>> scons: done reading SConscript files.
>>> scons: Cleaning targets ...
>>> scons: done cleaning targets.
>>> hua at grass:~/git/usw $
>>>
>>> BR, Yanghao
>>>
>>> On Sat, Jul 22, 2017 at 8:42 AM, Hua Yanghao <huayanghao at gmail.com> wrote:
>>> > Hi Bill,
>>> > No error seen if no "-j8".
>>> > I do have quite some custom builders, any special consideration needed?
>>> >
>>> > without -j8 using --tree=prune just builds normally:
>>> > hua at grass:~/git/usw $ tools/scons --tree=prune -c
>>> > --taskmastertrace=test.log config=linux64_full.py
>>> > scons: Reading SConscript files ...
>>> > scons: done reading SConscript files.
>>> > scons: Cleaning targets ...
>>> > scons: done cleaning targets.
>>> > hua at grass:~/git/usw $ tools/scons -j8 --tree=prune -c
>>> > --taskmastertrace=test.log config=linux64_full.py
>>> > scons: Reading SConscript files ...
>>> > scons: done reading SConscript files.
>>> > scons: Cleaning targets ...
>>> > scons: done cleaning targets.
>>> >
>>> > scons: *** Found dependency cycle(s):
>>> >   Internal Error: no cycle found for node output/linux64_full
>>> > (<SCons.Node.FS.Dir object at 0x564ad74aafe0>) in state pending
>>> >
>>> > File "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/Taskmaster.py",
>>> > line 1043, in cleanup
>>> > hua at grass:~/git/usw $
>>> >
>>> > Best Regards,
>>> > Yanghao
>>> >
>>> > On Sat, Jul 22, 2017 at 12:40 AM, Bill Deegan
>>> > <bill at baddogconsulting.com> wrote:
>>> >> If you build with no "-j8" do you see the error?
>>> >> If not then what's the output of --tree=prune without -j8?
>>> >>
>>> >> Also you didn't respond to:
>>> >>
>>> >> Do you have any custom Builders?
>>> >>
>>> >> -Bill
>>> >>
>>> >> On Fri, Jul 21, 2017 at 6:28 PM, Hua Yanghao <huayanghao at gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> hua at grass:~ $ python --version
>>> >>> Python 2.7.13
>>> >>> hua at grass:~ $ lsb_release -a
>>> >>> LSB Version:
>>> >>>
>>> >>> core-9.20160110ubuntu5-amd64:core-9.20160110ubuntu5-noarch:security-9.20160110ubuntu5-amd64:security-9.20160110ubuntu5-noarch
>>> >>> Distributor ID: Ubuntu
>>> >>> Description: Ubuntu 17.04
>>> >>> Release: 17.04
>>> >>> Codename: zesty
>>> >>> hua at grass:~ $
>>> >>>
>>> >>> I attached the taskmastertrace log here.
>>> >>>
>>> >>> --tree=prune shows the same result:
>>> >>> hua at grass:~/git/usw $ tools/scons -j8 --tree=prune -c
>>> >>> --taskmastertrace=test.log config=linux64_full.py
>>> >>> scons: Reading SConscript files ...
>>> >>> scons: done reading SConscript files.
>>> >>> scons: Cleaning targets ...
>>> >>> scons: done cleaning targets.
>>> >>>
>>> >>> scons: *** Found dependency cycle(s):
>>> >>>   Internal Error: no cycle found for node output/linux64_full
>>> >>> (<SCons.Node.FS.Dir object at 0x55d1fef0ffe0>) in state pending
>>> >>>
>>> >>> File "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/Taskmaster.py",
>>> >>> line 1043, in cleanup
>>> >>>
>>> >>> On Sat, Jul 22, 2017 at 12:03 AM, Bill Deegan
>>> >>> <bill at baddogconsulting.com>
>>> >>> wrote:
>>> >>> > Which version of Python?
>>> >>> > What operating system? (mac, linux, windows?)
>>> >>> >
>>> >>> > Do you have any custom Builders?
>>> >>> >
>>> >>> > Try using
>>> >>> > --taskmastertrace=file
>>> >>> >
>>> >>> > Prints trace information to the specified file about how the
>>> >>> > internal
>>> >>> > Taskmaster object evaluates and controls the order in which Nodes
>>> >>> > are
>>> >>> > built.
>>> >>> > A file name of - may be used to specify the standard output.
>>> >>> >
>>> >>> > And pastebin the output.
>>> >>> >
>>> >>> > Also anything unusual in --tree=prune ?
>>> >>> >
>>> >>> > -Bill
>>> >>> >
>>> >>> >
>>> >>> > On Fri, Jul 21, 2017 at 5:17 PM, Hua Yanghao <huayanghao at gmail.com>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> Dear Scons User,
>>> >>> >> I am recently experiencing some random failures in both scons build
>>> >>> >> and scons clean (-c).
>>> >>> >>
>>> >>> >> Unfortunately the project has more than 2000 .c/.h files and I
>>> >>> >> cannot
>>> >>> >> create a minimal project to reproduce this issue.
>>> >>> >>
>>> >>> >> Error message: (scons -c)
>>> >>> >> Removed output/linux64_full/libusw.a
>>> >>> >> Removed output/linux64_full/cmdlist.txt
>>> >>> >> scons: done cleaning targets.
>>> >>> >>
>>> >>> >> scons: *** Found dependency cycle(s):
>>> >>> >>   Internal Error: no cycle found for node output/linux64_full
>>> >>> >> (<SCons.Node.FS.Dir object at 0x5630bcf78c60>) in state pending
>>> >>> >>
>>> >>> >> File
>>> >>> >> "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/Taskmaster.py",
>>> >>> >> line 1043, in cleanup
>>> >>> >>
>>> >>> >> End of error message.
>>> >>> >>
>>> >>> >> This issue can be reproduced one out of five runs, could happen in
>>> >>> >> both scons build and scons clean.
>>> >>> >> This issue happens *only* with parallel builds (-j8), and cannot
>>> >>> >> reproduce with single thread build.
>>> >>> >> The project uses scons VariantDir() and many custom builders.
>>> >>> >>
>>> >>> >> In file SCons/Taskmaster.py, if I simply add a "return" on line
>>> >>> >> 1033,
>>> >>> >> the problem cannot be reproduced anymore, and builds are fine all
>>> >>> >> the
>>> >>> >> time (checked many dozens of times):
>>> >>> >> 1033         return # <--------- This line "fixed" the issue ;-)
>>> >>> >> 1034         desc = 'Found dependency cycle(s):\n'
>>> >>> >> 1035         for node, cycle in nclist:
>>> >>> >> 1036             if cycle:
>>> >>> >> 1037                 desc = desc + "  " + " -> ".join(map(str,
>>> >>> >> cycle))
>>> >>> >> +
>>> >>> >> "\n"
>>> >>> >> 1038             else:
>>> >>> >> 1039                 desc = desc + \
>>> >>> >> 1040                     "  Internal Error: no cycle found for node
>>> >>> >> %s
>>> >>> >> (%s) in state %s\n" %  \
>>> >>> >> 1041                     (node, repr(node),
>>> >>> >> StateString[node.get_state()])
>>> >>> >> 1042
>>> >>> >> 1043         raise SCons.Errors.UserError(desc)
>>> >>> >>
>>> >>> >> Does anyone had similar experience? Is scons over-checking
>>> >>> >> something
>>> >>> >> and made a false alarm?
>>> >>> >> I am about to promote scons to a much wider audience but this issue
>>> >>> >> really makes me a little bit nervous.
>>> >>> >>
>>> >>> >> Thanks a lot for your help,
>>> >>> >> Yanghao
>>> >>> >> _______________________________________________
>>> >>> >> Scons-users mailing list
>>> >>> >> Scons-users at scons.org
>>> >>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > _______________________________________________
>>> >>> > Scons-users mailing list
>>> >>> > Scons-users at scons.org
>>> >>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>> >
>>> >>>
>>> >>> _______________________________________________
>>> >>> Scons-users mailing list
>>> >>> Scons-users at scons.org
>>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Scons-users mailing list
>>> >> Scons-users at scons.org
>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>