[Scons-users] Fwd:Re: timing issues and protecting from them

Wed Dec 9 11:22:24 EST 2015

From: Tom Tanner (BLOOMBERG/ LONDON) At: Dec  9 2015 16:22:08
To: dl9obn at darc.de
Subject: Fwd:Re: [Scons-users] timing issues and protecting from them

I don't think this is a problem. 

1) before the build you have a set of source dependencies with timestamps
2) You do the build of the target.
3) You then compare the timestamps
4) Assuming they haven't changed, you save the result.

There is no need to have a correct order. If *any* timestamp has changed between deciding the target needed building and finishing the check of all the timestamps, then you have a potential problem. OK, you might get a false positive if a file timestamp changed right at the end. But you won't get a false negative.

Also, the issue is NOT that you have a library. The point is that

1) You have a library that depends on a.o, b.o, c.o
2) a.o depends on a.cc, a.hh, b.hh; etc etc

So when you build a.o, before you decide the build was succesful, you check the times on a.cc, a.hh, b.hh

Then when you build b.o, you do the same sort of thing.

When you build z, you can check the .os if you're feeling (a) paranoid or (b) it makes the coding easier.

At that point though, it doesn't matter if a.hh has been changed because then the next build will correctly deduce that a.o is out of date.

So I believe this does guarantee the build is consistent and correct, because you build an object then check the timestamps of what immediately contribute to the object haven't changed. Per object, not over the whole build. There isn't a correct order issue because you do this check per object built, not after the whole build.

Repeat for your example

Z depends on A, B, C, D

So you stash md5 and timestamp of A, B, C, D

build Z

check A, B, C, D in any order you like

If A or B or C or D has changed, the build of Z has failed

Otherwise you store Z as a succesful build with the md5/timestamp of A/B/C/D as all is OK, as they were the same before and after you built Z

(pedant: unless someone changed A and then changed it back including restoring the timestamp).

From: dl9obn at darc.de At: Dec  9 2015 14:57:30
To: scons-users at scons.org
Subject: Re: [Scons-users] timing issues and protecting from them

Hi Tom,

On 09.12.2015 14:15, Tom Tanner (BLOOMBERG/ LONDON) wrote:
> The problem is not in the generated file. It's in the source file.
>
> It's fine to change a source file outside scons (I hope). Just doing it while 
a build is in process is opening you up to race hazards.
>

but your approach of checking the timestamps of source files (or their MD5 
sums, or anything else) won't be able to guarantee a 
self-consistent and correct build either:

Let's say we have a target (library) Z, which depends on several sources A, B, 
C, D. So, after Z is built() you start to loop 
through the source dependencies in their lexicographical order. This checking 
will take time, and even if the single checks are 
relatively fast, it can still happen that you're currently checking C (with D 
still to go) and suddenly someone changes A from the 
outside. Then you get the same problem as before, an inconsistent state of 
timestamps vs hash.
Note how this problem is independent of which order you choose for processing 
the single children. You simply don't know the 
"correct" order beforehand...the one that you'd have to use for traversing the 
graph, such that the final modification happens 
before checking the corresponding source again.

So, from my perspective it looks as if guaranteeing that the build is 
self-consistent and correct, while allowing modifications from 
the outside, is impossible to do. What your approach definitely delivers is a 
slighter chance for a mishap like this...but I'm 
frankly more interested in solutions that give us the full 100% all the time 
(including parallel builds, which we haven't even 
touched yet ;)) instead of a 99.997% (I totally made that number up, but you 
get the idea).

Best regards,

Dirk

> From: wblevins001 at gmail.com At: Dec 9 2015 09:58:20
> To: Tom Tanner (BLOOMBERG/ LONDON) <mailto:ttanner2 at bloomberg.net>, 
scons-users at scons.org <mailto:scons-users at scons.org>
> Subject: Re: [Scons-users] timing issues and protecting from them
>
>     Read inline comments.
>
>     On Wed, Dec 9, 2015 at 8:11 AM, Tom Tanner (BLOOMBERG/ LONDON) 
<ttanner2 at bloomberg.net <mailto:ttanner2 at bloomberg.net>> wrote:
>
>         No. I'm implying it should happen at the time it does the build. Not 
during subsequent runs.
>
>         The situation we got into is:
>
>         timestamp in scons.dblite matches current timestamp of file
>         md5 in scons.dblite doesn't match current md5 of file
>
>         We have a somewhat paranoid decider function which checks for 
mismatchming md5s but that doesn't trigger if the timestamp
>         hasn't changed (as otherwise it would be very slow).
>
>         So the check I'm thinking of is something like in the .built function
>
>         if database timestamp of source dependency != actual timestamp of 
source dependency:
>         raise builderror (str(source) + ' changed during build')
>
>

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20151209/ee3d9e70/attachment-0001.html>