[Scons-users] Fwd:Re: timing issues and protecting from them

Wed Dec 9 17:02:41 EST 2015

Tom,

I'm guessing you just took a big pain point tracing down this sequence of
events.
Likely not an intentional error.
But due to someone's incompetence (perhaps a strong word, but you don't do
this if you know what you are doing and it's possible effects)

Does this happen frequently in your organization?  Is it usually just one
individual who's the cause?
Or is this a one-off which was very painful?

I can understand your interest in ensuring that this doesn't happen again,
but I think the summary of all the responses are:
1) This seems like it would be expensive time wise
2) This seems difficult to actually implement correctly and test
(especially when you throw in parallel builds)
3) This was caused by someone's mistake and could have been

Every now and then someone pops up with a similar issue, but rarely with a
patch and tests to go with it. (I believe once a year or less)

The big question mark for me is does it make sense to include such
functionality in SCons?
I'm usually inclined to just suggest when a user interferes with a build
and causes such to just tell them not to do it again.
Part of me feels this much sanity checking may be out of scope (for SCons)?
(And I fully admit I can be 100% wrong on this)

I wonder if there might be a way to do this as a PostAction()..

-Bill

On Wed, Dec 9, 2015 at 3:43 PM, Tom Tanner <trtanner at btinternet.com> wrote:

> On 9/12/15 17:00, Dirk Bächle wrote:
>
>> Tom,
>>
>> On 09.12.2015 17:22, Tom Tanner (BLOOMBERG/ LONDON) wrote:
>>
>>>
>>> [...]
>>>
>>>     I don't think this is a problem.
>>>
>>>
>> okay so let's go with your example then...
>>
>> [...]
>>>
>>>     There is no need to have a correct order. If *any* timestamp has
>>> changed between deciding the target needed building and
>>>     finishing the check of all the timestamps, then you have a potential
>>> problem. OK, you might get a false positive if a file
>>>     timestamp changed right at the end. But you won't get a false
>>> negative.
>>>
>>>
>> My point here is that you're not always able to detect that the timestamp
>> actually changed, based on the fact that you have to choose a processing
>> order. And this order then might interfere with what's actually happening...
>>
>>     Also, the issue is NOT that you have a library. The point is that
>>>
>>>     1) You have a library that depends on a.o, b.o, c.o
>>>     2) a.o depends on a.cc, a.hh, b.hh; etc etc
>>>
>>>     So when you build a.o, before you decide the build was succesful,
>>> you check the times on a.cc, a.hh, b.hh
>>>
>>>
>> So, you have checked the times of "a.cc" and "a.hh" already, and are
>> about to process "b.hh". Now, suddenly someone from the outside changes
>> "a.cc", and you won't detect this timestamp change. Your build is
>> inconsistent, depending on how you define "consistency".
>> You might say: That's a case for the next build, because the file was
>> changed *after* checking. But then what's your "horizon" (can't think of a
>> better word right now) for checking. Until when is it allowed to change a
>> file, and when do you say: "That change has to be handled by the next
>> build"?
>>
>> This is why I say - have any of the sources changed *while* I was
> building the target. This you check *after* the target has been built. So
> the database currently says
>
> a.hh - md5 MAH, time TAH
> a.cc - md5 MAC, time TAC
> b.hh - md5 MBH, time TBH
>
> then you execute the build actions to generate a.o
>
> then you check if any of a.hh, a.cc, b.hh have changed (on disk) from the
> values you currently have stored.
>
> And if they have, then you no longer have the right information to
> determine what the pre-requisites of a.o are, because you don't know when
> whatever changed did change. The prerequisite could have changed before the
> build action used the contents or it could have changed after. And if you
> pick the wrong one, you'll get incorrect information stored. And you
> shouldn't in any case store the changed time for a.hh because that could
> break the database for other targets that you've already built that also
> depend on a.hh. So the only thing you can do is generate an error.
>
> If however, a.hh changes *after* this point, you don't need to worry
> because you know the next build will identify the fact that a.hh is
> different when it evaluates whether or not it needs to build a.o, because
> you will have stored the correct information about the state of the
> prerequisites anyway.
>
> So basically your procedure for building seems to need to be something
> like:
>
>     I have target T to build from S1, S2, S3
>
>    Each Si has a timestamp and md5sum. There is the one from the previous
> build and the one we checked when re-evaluating the tree. Both these are
> immutable once calculated.
>
>    Do T's build action
>
>    Check each Si to see if it has changed. If so generate an error.
>
>    Store T's pre-requisites. These will refer to the immutable timestamps.
>
> There is no horizon you have to worry about. Either the file is different
> between before the build action and after the build action, in which case
> you have a problem, or it isn't, in which case you know the build was done
> with the versions of the files you think it was built with and all is OK.
>
> It doesn't matter if you then do 0, 5, 10, 500 or even 5000 other build
> actions and change a.hh at some point during that (and presuming they don't
> depend on a.hh either!). Because T says 'I depend on a.hh with xxx
> timestamp/md5' and next time you do a build, scons will see that a.hh has
> changed.
>
> But this sequence will not work, which is what you seem to be suggesting
> is happening:
>
>     get time/md5 of a.hh, b.hh, a.cc
>     do build action
>     check a.hh
>     check b.hh
>     check a.cc
>     <---- a.hh changes externally
>     store *current on disk* information of a.hh, b.hh, a.cc (with the new
> timestamp of a.hh)
>
>  If that is what happens, that's going to go horribly wrong.
>
> Basically I suppose you're allowed to change a file right up until scons
> has looked at it and decided it has changed, and then you shouldn't change
> it until scons has built everything that directly depends on it.
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20151209/1ec75342/attachment.html>