[Scons-users] scons-local-2.5.1 build and clean shows random failure

Bill Deegan bill at baddogconsulting.com
Tue Jul 25 10:43:31 EDT 2017 

I need a taskmaster trace from a parallel build which fails to be able to
see more about what's happening.

A single thread build after a failed build doesn't yield any useful
information.

As a data point this issue is not that widespread as parallel build is used
widely without issue.

Are you using SideEffects()?

-Bill


On Tue, Jul 25, 2017 at 1:52 AM, Hua Yanghao <huayanghao at gmail.com> wrote:

> Any one has any new ideas? Seems this is a quite wide spread issue,
> scons seems are not handling the cyclic dependency reliable enough on
> parallel build.
>
> On Sat, Jul 22, 2017 at 7:49 PM, Hua Yanghao <huayanghao at gmail.com> wrote:
> > Yes, I trigger the single build when it fails, and generated the file.
> > Just in case I generated it again and attached here.
> >
> > On Sat, Jul 22, 2017 at 6:08 PM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
> >> Is the taskmaster trace file from when you saw a failure?
> >>
> >> On Fri, Jul 21, 2017 at 11:47 PM, Hua Yanghao <huayanghao at gmail.com>
> wrote:
> >>>
> >>> By the way the failure is random, below 4 consecutive runs failed
> >>> twice passed twice:
> >>> hua at grass:~/git/usw $ tools/scons -j8 -c config=linux64_full.py
> >>> scons: Reading SConscript files ...
> >>> scons: done reading SConscript files.
> >>> scons: Cleaning targets ...
> >>> scons: done cleaning targets.
> >>> hua at grass:~/git/usw $ tools/scons -j8 -c config=linux64_full.py
> >>> scons: Reading SConscript files ...
> >>> scons: done reading SConscript files.
> >>> scons: Cleaning targets ...
> >>> scons: done cleaning targets.
> >>>
> >>> scons: *** Found dependency cycle(s):
> >>>   Internal Error: no cycle found for node output/linux64_full
> >>> (<SCons.Node.FS.Dir object at 0x55d10f3ee5a0>) in state pending
> >>>
> >>> File "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/Taskmaster.py",
> >>> line 1043, in cleanup
> >>> hua at grass:~/git/usw $ tools/scons -j8 -c config=linux64_full.py
> >>> scons: Reading SConscript files ...
> >>> scons: done reading SConscript files.
> >>> scons: Cleaning targets ...
> >>> scons: done cleaning targets.
> >>>
> >>> scons: *** Found dependency cycle(s):
> >>>   Internal Error: no cycle found for node output/linux64_full
> >>> (<SCons.Node.FS.Dir object at 0x55f9ed59ef20>) in state pending
> >>>
> >>> File "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/Taskmaster.py",
> >>> line 1043, in cleanup
> >>> hua at grass:~/git/usw $ tools/scons -j8 -c config=linux64_full.py
> >>> scons: Reading SConscript files ...
> >>> scons: done reading SConscript files.
> >>> scons: Cleaning targets ...
> >>> scons: done cleaning targets.
> >>> hua at grass:~/git/usw $
> >>>
> >>> BR, Yanghao
> >>>
> >>> On Sat, Jul 22, 2017 at 8:42 AM, Hua Yanghao <huayanghao at gmail.com>
> wrote:
> >>> > Hi Bill,
> >>> > No error seen if no "-j8".
> >>> > I do have quite some custom builders, any special consideration
> needed?
> >>> >
> >>> > without -j8 using --tree=prune just builds normally:
> >>> > hua at grass:~/git/usw $ tools/scons --tree=prune -c
> >>> > --taskmastertrace=test.log config=linux64_full.py
> >>> > scons: Reading SConscript files ...
> >>> > scons: done reading SConscript files.
> >>> > scons: Cleaning targets ...
> >>> > scons: done cleaning targets.
> >>> > hua at grass:~/git/usw $ tools/scons -j8 --tree=prune -c
> >>> > --taskmastertrace=test.log config=linux64_full.py
> >>> > scons: Reading SConscript files ...
> >>> > scons: done reading SConscript files.
> >>> > scons: Cleaning targets ...
> >>> > scons: done cleaning targets.
> >>> >
> >>> > scons: *** Found dependency cycle(s):
> >>> >   Internal Error: no cycle found for node output/linux64_full
> >>> > (<SCons.Node.FS.Dir object at 0x564ad74aafe0>) in state pending
> >>> >
> >>> > File "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/
> Taskmaster.py",
> >>> > line 1043, in cleanup
> >>> > hua at grass:~/git/usw $
> >>> >
> >>> > Best Regards,
> >>> > Yanghao
> >>> >
> >>> > On Sat, Jul 22, 2017 at 12:40 AM, Bill Deegan
> >>> > <bill at baddogconsulting.com> wrote:
> >>> >> If you build with no "-j8" do you see the error?
> >>> >> If not then what's the output of --tree=prune without -j8?
> >>> >>
> >>> >> Also you didn't respond to:
> >>> >>
> >>> >> Do you have any custom Builders?
> >>> >>
> >>> >> -Bill
> >>> >>
> >>> >> On Fri, Jul 21, 2017 at 6:28 PM, Hua Yanghao <huayanghao at gmail.com>
> >>> >> wrote:
> >>> >>>
> >>> >>> hua at grass:~ $ python --version
> >>> >>> Python 2.7.13
> >>> >>> hua at grass:~ $ lsb_release -a
> >>> >>> LSB Version:
> >>> >>>
> >>> >>> core-9.20160110ubuntu5-amd64:core-9.20160110ubuntu5-noarch:
> security-9.20160110ubuntu5-amd64:security-9.20160110ubuntu5-noarch
> >>> >>> Distributor ID: Ubuntu
> >>> >>> Description: Ubuntu 17.04
> >>> >>> Release: 17.04
> >>> >>> Codename: zesty
> >>> >>> hua at grass:~ $
> >>> >>>
> >>> >>> I attached the taskmastertrace log here.
> >>> >>>
> >>> >>> --tree=prune shows the same result:
> >>> >>> hua at grass:~/git/usw $ tools/scons -j8 --tree=prune -c
> >>> >>> --taskmastertrace=test.log config=linux64_full.py
> >>> >>> scons: Reading SConscript files ...
> >>> >>> scons: done reading SConscript files.
> >>> >>> scons: Cleaning targets ...
> >>> >>> scons: done cleaning targets.
> >>> >>>
> >>> >>> scons: *** Found dependency cycle(s):
> >>> >>>   Internal Error: no cycle found for node output/linux64_full
> >>> >>> (<SCons.Node.FS.Dir object at 0x55d1fef0ffe0>) in state pending
> >>> >>>
> >>> >>> File "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/
> Taskmaster.py",
> >>> >>> line 1043, in cleanup
> >>> >>>
> >>> >>> On Sat, Jul 22, 2017 at 12:03 AM, Bill Deegan
> >>> >>> <bill at baddogconsulting.com>
> >>> >>> wrote:
> >>> >>> > Which version of Python?
> >>> >>> > What operating system? (mac, linux, windows?)
> >>> >>> >
> >>> >>> > Do you have any custom Builders?
> >>> >>> >
> >>> >>> > Try using
> >>> >>> > --taskmastertrace=file
> >>> >>> >
> >>> >>> > Prints trace information to the specified file about how the
> >>> >>> > internal
> >>> >>> > Taskmaster object evaluates and controls the order in which Nodes
> >>> >>> > are
> >>> >>> > built.
> >>> >>> > A file name of - may be used to specify the standard output.
> >>> >>> >
> >>> >>> > And pastebin the output.
> >>> >>> >
> >>> >>> > Also anything unusual in --tree=prune ?
> >>> >>> >
> >>> >>> > -Bill
> >>> >>> >
> >>> >>> >
> >>> >>> > On Fri, Jul 21, 2017 at 5:17 PM, Hua Yanghao <
> huayanghao at gmail.com>
> >>> >>> > wrote:
> >>> >>> >>
> >>> >>> >> Dear Scons User,
> >>> >>> >> I am recently experiencing some random failures in both scons
> build
> >>> >>> >> and scons clean (-c).
> >>> >>> >>
> >>> >>> >> Unfortunately the project has more than 2000 .c/.h files and I
> >>> >>> >> cannot
> >>> >>> >> create a minimal project to reproduce this issue.
> >>> >>> >>
> >>> >>> >> Error message: (scons -c)
> >>> >>> >> Removed output/linux64_full/libusw.a
> >>> >>> >> Removed output/linux64_full/cmdlist.txt
> >>> >>> >> scons: done cleaning targets.
> >>> >>> >>
> >>> >>> >> scons: *** Found dependency cycle(s):
> >>> >>> >>   Internal Error: no cycle found for node output/linux64_full
> >>> >>> >> (<SCons.Node.FS.Dir object at 0x5630bcf78c60>) in state pending
> >>> >>> >>
> >>> >>> >> File
> >>> >>> >> "/home/hua/git/usw/tools/scons-local-2.5.1/SCons/
> Taskmaster.py",
> >>> >>> >> line 1043, in cleanup
> >>> >>> >>
> >>> >>> >> End of error message.
> >>> >>> >>
> >>> >>> >> This issue can be reproduced one out of five runs, could happen
> in
> >>> >>> >> both scons build and scons clean.
> >>> >>> >> This issue happens *only* with parallel builds (-j8), and cannot
> >>> >>> >> reproduce with single thread build.
> >>> >>> >> The project uses scons VariantDir() and many custom builders.
> >>> >>> >>
> >>> >>> >> In file SCons/Taskmaster.py, if I simply add a "return" on line
> >>> >>> >> 1033,
> >>> >>> >> the problem cannot be reproduced anymore, and builds are fine
> all
> >>> >>> >> the
> >>> >>> >> time (checked many dozens of times):
> >>> >>> >> 1033         return # <--------- This line "fixed" the issue ;-)
> >>> >>> >> 1034         desc = 'Found dependency cycle(s):\n'
> >>> >>> >> 1035         for node, cycle in nclist:
> >>> >>> >> 1036             if cycle:
> >>> >>> >> 1037                 desc = desc + "  " + " -> ".join(map(str,
> >>> >>> >> cycle))
> >>> >>> >> +
> >>> >>> >> "\n"
> >>> >>> >> 1038             else:
> >>> >>> >> 1039                 desc = desc + \
> >>> >>> >> 1040                     "  Internal Error: no cycle found for
> node
> >>> >>> >> %s
> >>> >>> >> (%s) in state %s\n" %  \
> >>> >>> >> 1041                     (node, repr(node),
> >>> >>> >> StateString[node.get_state()])
> >>> >>> >> 1042
> >>> >>> >> 1043         raise SCons.Errors.UserError(desc)
> >>> >>> >>
> >>> >>> >> Does anyone had similar experience? Is scons over-checking
> >>> >>> >> something
> >>> >>> >> and made a false alarm?
> >>> >>> >> I am about to promote scons to a much wider audience but this
> issue
> >>> >>> >> really makes me a little bit nervous.
> >>> >>> >>
> >>> >>> >> Thanks a lot for your help,
> >>> >>> >> Yanghao
> >>> >>> >> _______________________________________________
> >>> >>> >> Scons-users mailing list
> >>> >>> >> Scons-users at scons.org
> >>> >>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>> >
> >>> >>> >
> >>> >>> >
> >>> >>> > _______________________________________________
> >>> >>> > Scons-users mailing list
> >>> >>> > Scons-users at scons.org
> >>> >>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>> >
> >>> >>>
> >>> >>> _______________________________________________
> >>> >>> Scons-users mailing list
> >>> >>> Scons-users at scons.org
> >>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>>
> >>> >>
> >>> >>
> >>> >> _______________________________________________
> >>> >> Scons-users mailing list
> >>> >> Scons-users at scons.org
> >>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>
> >>> _______________________________________________
> >>> Scons-users mailing list
> >>> Scons-users at scons.org
> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>
> >>
> >>
> >> _______________________________________________
> >> Scons-users mailing list
> >> Scons-users at scons.org
> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170725/0c322cdf/attachment-0001.html>

More information about the Scons-users mailing list