[Scons-users] Fwd:Re: timing issues and protecting from them
Tom Tanner (BLOOMBERG/ LONDON)
ttanner2 at bloomberg.net
Fri Dec 11 11:23:18 EST 2015
This is the change
+ import os
+ import stat
def executed_with_callbacks(self):
"""
Called when the task has been successfully executed and
the Taskmaster instance wants to call the Node's callback
methods.
This may have been a do-nothing operation (to preserve build
order), so we must check the node's state before deciding whether
it was "built", in which case we call the appropriate Node method.
In any event, we always call "visited()", which will handle any
post-visit actions that must take place regardless of whether
or not the target was an actual built target or a source Node.
"""
T = self.tm.trace
if T: T.write(self.trace_message('Task.executed_with_callbacks()',
self.node))
for t in self.targets:
if t.get_state() == NODE_EXECUTING:
for side_effect in t.side_effects:
side_effect.set_state(NODE_NO_STATE)
t.set_state(NODE_EXECUTED)
# ++++++++++++++++ ADDED CODE
for s in t.depends + t.executor.get_all_children():
if s.rexists():
ok = True
try:
s1 = s.get_timestamp()
s2 = os.stat(s.rfile().absfile)[stat.ST_MTIME]
if s1 != s2:
ok = False
except:
pass
if not ok:
raise SCons.Errors.Builderror(errstr = s.str() + ' changed during build')
# ++++++++++++++++++++++++++++
if not t.cached:
t.cached = t.push_to_cache()
t.built()
t.visited()
I haven't run the tests as I'm running this in a real build for now
From: wblevins001 at gmail.com At: Dec 11 2015 14:33:47
To: Tom Tanner (BLOOMBERG/ LONDON), scons-users at scons.org
Subject: Re: [Scons-users] Fwd:Re: timing issues and protecting from them
Can you post of diff snippet, so we have some context? Do the tests still pass?
V/R,
William
On Fri, Dec 11, 2015 at 1:14 PM, Tom Tanner (BLOOMBERG/ LONDON) <ttanner2 at bloomberg.net> wrote:
Having fixed some problems in that (like getmtime() being cached), I've found that doing this even when copying files out of the cache (i.e. putting the for loop *before* "if not t.cached: " actually speeds up the build noticeably.
Running -j2 on a linux box:
without 'change' test: Total build time: 950-1350s.
with 'change' test : Total build time: 950-1060s
Given this is a multi-user machine I'm not really sure how much I trust the times but it seems the overhead of doing this check is well below the overhead of other things going on.
From: ttanner2 at bloomberg.net At: Dec 10 2015 17:49:46
To: bill at baddogconsulting.com, scons-users at scons.org
Subject: Re: [Scons-users] Fwd:Re: timing issues and protecting from them
Presuming this is all one build:
I think I'm correct in stating that the calculation of the md5 and timestamp of libabc_api.h is done once, before any of a.o, b.o, x.o are built.
So the (proposed) system would say
Decide we need libabc_api.h, cache the timestamp and md5
a.o - built, validates against timestamps
b.o - built, validates against timestamps
libabc_api.h changes
libabc_abi.lib built, validates against .o timestamps
x.o - built,
libapc_api.h cached timestamp != libapc_api.h actual timestamp
error
In passing, I'm poking around with the following implementation of Task.executed_with_callbacks
def executed_with_callbacks(self):
"""
Called when the task has been successfully executed and
the Taskmaster instance wants to call the Node's callback
methods.
This may have been a do-nothing operation (to preserve build
order), so we must check the node's state before deciding whether
it was "built", in which case we call the appropriate Node method.
In any event, we always call "visited()", which will handle any
post-visit actions that must take place regardless of whether
or not the target was an actual built target or a source Node.
"""
T = self.tm.trace
if T: T.write(self.trace_message('Task.executed_with_callbacks()',
self.node))
for t in self.targets:
if t.get_state() == NODE_EXECUTING:
for side_effect in t.side_effects:
side_effect.set_state(NODE_NO_STATE)
t.set_state(NODE_EXECUTED)
if not t.cached:
# ++++++++++++++++ ADDED CODE
for s in t.depends + t.executor.get_all_children():
if s.rexists():
ok = True
try:
s1 = s.get_timestamp()
s2 = s.rfile().getmtime()
if s1 != s2:
ok = False
except:
pass
if not ok:
raise SCons.Errors.Builderror(s.str() + ' changed during build')
# ++++++++++++++++++++++++++++
t.cached = t.push_to_cache()
t.built()
t.visited()
executed_without_callback seems to be only used for clean, and I don't think this test would apply to clean!
Seems to be about enough. Not sure about the cost as it's only going to activate if you actually do a build without copying from cache, which I'd think is quite expensive in any case, so I shall have to bite the bullet and try a full build without using the cache.
From: bill at baddogconsulting.com At: Dec 10 2015 14:11:57
To: Tom Tanner (BLOOMBERG/ LONDON), scons-users at scons.org
Subject: Re: [Scons-users] Fwd:Re: timing issues and protecting from them
Tom,
How about this sequence.
a.o, b.o built agains libabc_api.h
libabc_api.h is modified in a way that breaks.. changing an enum for example
libabc_abi.lib built
x.o compiles and uses libabc_api.h
Your compile finishes
Now your code my break in strange and wonderous ways..
-Bill
On Thu, Dec 10, 2015 at 8:42 AM, Tom Tanner (BLOOMBERG/ LONDON) <ttanner2 at bloomberg.net> wrote:
Thanks. I'll start poking around in those places.
And yes, an indirect dependency can safely change because it isn't part of your signature. Only the direct dependencies are.
Consider:
a,o depends on a.hh, a.cc
a.lib depends on a.o
scons starts build
scons builds a.o and updates tree
idiot^H^H^H^H^Huser changes a.hh
scons build a.lib
You don't have an inconsistency. You do have a requirement to rebuild, but nothing is inconsistent.
From: dl9obn at darc.de At: Dec 10 2015 10:39:05To: scons-users at scons.org
Subject: Re: [Scons-users] Fwd:Re: timing issues and protecting from them
Tom,
On 10.12.2015 09:13, Tom Tanner (BLOOMBERG/ LONDON) wrote:
> Well, I can check the timing for that. But I don't see a necessity for
rechecking the indirect dependencies
>
so an indirect dependency may change during a build, but a direct source not?
This doesn't really make sense to me...but I
understand where you're coming from. You have experienced this phenomenon of
"hard to track down" build errors several times now,
and want to do something about it. So please, as I mentioned in my earlier
mail, start on an implementation for this feature if you
find the time. The execute() and executed_with_callbacks() methods in the
Taskmaster look like the best place to do that.
To me, this seems more like a job for the Taskmaster than putting new code in
the Node() methods build()/built().
I'd derive a new ParanoidTaskmaster from the original class and then add my
checks to that. This also includes generalizing the
creation of the "Taskmaster" in the Main.py script, such that you can select
between the "default" and your new version via a
command-line switch.
Finally, the really paranoid people clearly separate builds from their edit
cycles by building the final stuff on a different server
for example. I happen to work in one of those places ;)...and we always build
against fixed labels.
Best regards,
Dirk
_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20151211/1d1f1d70/attachment-0001.html>
More information about the Scons-users
mailing list