[Scons-users] Fwd:Re: timing issues and protecting from them

Fri Dec 11 09:33:45 EST 2015

Can you post of diff snippet, so we have some context? Do the tests still
pass?

V/R,
William

On Fri, Dec 11, 2015 at 1:14 PM, Tom Tanner (BLOOMBERG/ LONDON) <
ttanner2 at bloomberg.net> wrote:

> Having fixed some problems in that (like getmtime() being cached), I've
> found that doing this even when copying files out of the cache (i.e.
> putting the for loop *before* "if not t.cached: " actually speeds up the
> build noticeably.
>
> Running -j2 on a linux box:
> without 'change' test: Total build time: 950-1350s.
> with 'change' test : Total build time: 950-1060s
>
> Given this is a multi-user machine I'm not really sure how much I trust
> the times but it seems the overhead of doing this check is well below the
> overhead of other things going on.
>
>
> From: ttanner2 at bloomberg.net At: Dec 10 2015 17:49:46
> To: bill at baddogconsulting.com, scons-users at scons.org
>
> Subject: Re: [Scons-users] Fwd:Re: timing issues and protecting from them
>
> Presuming this is all one build:
>
> I think I'm correct in stating that the calculation of the md5 and
> timestamp of libabc_api.h is done once, before any of a.o, b.o, x.o are
> built.
>
> So the (proposed) system would say
> Decide we need libabc_api.h, cache the timestamp and md5
> a.o - built, validates against timestamps
> b.o - built, validates against timestamps
> libabc_api.h changes
> libabc_abi.lib built, validates against .o timestamps
> x.o - built,
> libapc_api.h cached timestamp != libapc_api.h actual timestamp
> error
>
>
> In passing, I'm poking around with the following implementation of
> Task.executed_with_callbacks
>
> def executed_with_callbacks(self):
> """
> Called when the task has been successfully executed and
> the Taskmaster instance wants to call the Node's callback
> methods.
>
> This may have been a do-nothing operation (to preserve build
> order), so we must check the node's state before deciding whether
> it was "built", in which case we call the appropriate Node method.
> In any event, we always call "visited()", which will handle any
> post-visit actions that must take place regardless of whether
> or not the target was an actual built target or a source Node.
> """
> T = self.tm.trace
> if T: T.write(self.trace_message('Task.executed_with_callbacks()',
> self.node))
>
> for t in self.targets:
> if t.get_state() == NODE_EXECUTING:
> for side_effect in t.side_effects:
> side_effect.set_state(NODE_NO_STATE)
> t.set_state(NODE_EXECUTED)
> if not t.cached:
> # ++++++++++++++++ ADDED CODE
> for s in t.depends + t.executor.get_all_children():
> if s.rexists():
> ok = True
> try:
> s1 = s.get_timestamp()
> s2 = s.rfile().getmtime()
> if s1 != s2:
> ok = False
> except:
> pass
> if not ok:
> raise SCons.Errors.Builderror(s.str() + ' changed during build')
> # ++++++++++++++++++++++++++++
> t.cached = t.push_to_cache()
> t.built()
> t.visited()
>
> executed_without_callback seems to be only used for clean, and I don't
> think this test would apply to clean!
>
> Seems to be about enough. Not sure about the cost as it's only going to
> activate if you actually do a build without copying from cache, which I'd
> think is quite expensive in any case, so I shall have to bite the bullet
> and try a full build without using the cache.
>
> From: bill at baddogconsulting.com At: Dec 10 2015 14:11:57
> To: Tom Tanner (BLOOMBERG/ LONDON) <#151912e2846c9e97_>,
> scons-users at scons.org <#151912e2846c9e97_>
> Subject: Re: [Scons-users] Fwd:Re: timing issues and protecting from them
>
> Tom,
>
> How about this sequence.
>
> a.o, b.o built agains libabc_api.h
> libabc_api.h is modified in a way that breaks.. changing an enum for
> example
> libabc_abi.lib built
> x.o compiles and uses libabc_api.h
> Your compile finishes
> Now your code my break in strange and wonderous ways..
>
> -Bill
>
> On Thu, Dec 10, 2015 at 8:42 AM, Tom Tanner (BLOOMBERG/ LONDON) <
> ttanner2 at bloomberg.net> wrote:
>
>> Thanks. I'll start poking around in those places.
>>
>> And yes, an indirect dependency can safely change because it isn't part
>> of your signature. Only the direct dependencies are.
>>
>> Consider:
>>
>> a,o depends on a.hh, a.cc
>> a.lib depends on a.o
>>
>> scons starts build
>> scons builds a.o and updates tree
>> idiot^H^H^H^H^Huser changes a.hh
>> scons build a.lib
>>
>> You don't have an inconsistency. You do have a requirement to rebuild,
>> but nothing is inconsistent.
>>
>>
>>
>> From: dl9obn at darc.de At: Dec 10 2015 10:39:05
>> To: scons-users at scons.org
>> Subject: Re: [Scons-users] Fwd:Re: timing issues and protecting from them
>>
>> Tom,
>>
>> On 10.12.2015 09:13, Tom Tanner (BLOOMBERG/ LONDON) wrote:
>> > Well, I can check the timing for that. But I don't see a necessity for
>> rechecking the indirect dependencies
>> >
>>
>>
>> so an indirect dependency may change during a build, but a direct source not?
>> This doesn't really make sense to me...but I
>>
>> understand where you're coming from. You have experienced this phenomenon of
>> "hard to track down" build errors several times now,
>>
>> and want to do something about it. So please, as I mentioned in my earlier
>> mail, start on an implementation for this feature if you
>> find the time. The execute() and executed_with_callbacks() methods in the
>> Taskmaster look like the best place to do that.
>>
>> To me, this seems more like a job for the Taskmaster than putting new code in
>> the Node() methods build()/built().
>>
>> I'd derive a new ParanoidTaskmaster from the original class and then add my
>> checks to that. This also includes generalizing the
>>
>> creation of the "Taskmaster" in the Main.py script, such that you can select
>> between the "default" and your new version via a
>> command-line switch.
>>
>>
>> Finally, the really paranoid people clearly separate builds from their edit
>> cycles by building the final stuff on a different server
>>
>> for example. I happen to work in one of those places ;)...and we always build
>> against fixed labels.
>>
>> Best regards,
>>
>> Dirk
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing listScons-users at scons.orghttps://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20151211/f02440d7/attachment.html>