[Scons-users] Another question about a strange tool

[Tom Tanner (BLOOMBERG/ LONDON)]

The problem is that Scons will copy the stuff from the cache and then purify detects that timestamps don't match so it repurifies everything.

it works something like this:

have libb.so -> libc.so

libb.so, libb.so_pure, libc.so, libc.so_pure in cache with timestamps B, B, C and C

you change something so that libc.so rebuilds but doesn't change. Purify however rebuilds libc.so_pur

so you have libc.so (C1) and libc.so_pur(C1)

when these get pushed to cache, libc.so is already there. so it doesn't get updated.

Next time you pull stuff from cache, you get

libc.so(C), libc.so_pur(C1)

if you now do something that causes libb.so to be rebuilt, purify will see libc.so and libc.so_pur timestamps don't match, and repurify it, giving you libc.so_pur(C, v2) with a different md5sum.

Now this wouldn't matter so much, except because libc.so_pur has now changed, it means that next time you do a full build, it'll spot that anything depending on the instrumented files (unit tests mainly) need rebuilding and rerunning. Some of our unit tests take a long time (and in some cases, merely purifying a library can take a while), so we like to have all the necessary stuff cached and reused if we can.

So far, I'm getting reasonable results by making the decider check if the timestamp of a purified file matches that of the timestamp of the source file, and if not, saying it needs a rebuild. And making the Action to generate the purified file include a line which calls a python function which returns the timestamp of the source, something like

[ '$PURIFY .... ', '@#$SOURCE_TIMESTAMP' ]

where env['SOURCE_TIMESTAMP'] is set to this:

def source_timetamp(target, source, env, for_signature):
return env.Value(source[0].get_timestamp())

If I could make a dependency on the source timestamp directly I would, but the emitter seems to be evaluated too early, and dependencies in the generator appear to be evaluated too late, and this seems to be the only way of getting this to work. But it seems a bit messy and I was wondering if there was a better way of doing this.

----- Original Message -----
From: bill at baddogconsulting.com
To: Tom Tanner (BLOOMBERG/ LONDON), scons-users at scons.org
At: Mar 18 2014 18:36:47

Tom,

Unless you have an NDA always better to state the tool.
Reasonable chance someone else will have used it and run into similar issue, and remember it better by the tool name, rather than thinking.. “huh.. that sounds familiar, but I can’t remember…”

I’m not certain how if a sub dependency changes, and that causes purify to run, would that be different than the Program() builder itself.

If progA -> libb.so -> libc.so, and libc.so rebuilds, won’t SCons rebuild the rest?

-Bill

On March 18, 2014 at 10:42:32 AM, Tom Tanner (BLOOMBERG/ LONDON) (ttanner2 at bloomberg.net) wrote:

> Yes, it is.

>  

> ----- Original Message -----

> From: bill at baddogconsulting.com

> To: Tom Tanner (BLOOMBERG/ LONDON), scons-users at scons.org

> At: Mar 18 2014 16:26:49

>  

> Tom,

>  

> Is the tool Purify? or one of it’s sister tools?

>  

> -Bill

>  

> On March 18, 2014 at 2:18:51 AM, Tom Tanner (BLOOMBERG/ LONDON) (ttanner2 at bloomberg.net)  

> wrote:

> > It's not a build tool. It's a tool that has its own build avoidance techniques. Which  

> are

> > exceedingly annoying when using it from a build system. We use it for code coverage and  

> > memory leak checks in unit tests which we automatically compile and run as part of the  

> > build.

> >

> > Basically the tool will create the target of its processing with an identical timestamp  

> > to the source. And it also checks for any shared libraries used.

> >

> > So if you run the tool on libA.so it creates libA.so_x with the same timestamp as libA.so  

> >

> > You then run this on libB.so which depends on libA.so. It creates libB.so_x with the  

> same

> > timestamp as libB.so, and then checks the timestamp of libA.so_x is the same as libA.so.  

> > If it isn't it will re-generate libA.so_x.

> >

> > Now consider if you rebuild libA.so for some reason but the contents don't change, so  

> > when this gets pushed to the cache, the object in the cache doesn't change. However when  

> > you run the tool it generates a different libA.so_x (it stores the timestamp in the generated  

> > file...) and that gets stored in the cache.

> >

> > Now do the build somewhere else using the cache.

> >

> > scons will pull in the original libA.so (with the old timestamp) and the new libA.so_x  

> > (with the new timestamp). This doesn't actually matter, until you actually change  

> something

> > that uses libA. Assume you change libB

> >

> > Then the tool starts to rebuild libB.so_x, and examines the dependencies. It finds  

> libA.so

> > and checks the timestamps. They don't match any more, so it builds libA.so_x.... Then  

> > you do another build and anything else that depended on libA.so_x will get rebuilt.  

> This

> > is not very stable.

> >

> > We have a custom decider which checks the md5 of an object is what it should be to protect  

> > against people manually targets with incorrect switches, testing, and submitting  

> > to production, which builds with different switches and can break (yes, this does happen!),  

> > and this behaviour of the tool and build triggers a mismatch as the libA.so_x has been  

> > changed.

> >

> > So basically, I'd like to add the timestamp of libA.so as a dependency of libA.so_x.  

> I've

> > done some experiments and it looks like I can't add dependencies in the generator because  

> > that's too late and causes continuous changes in the dependencies, and either I have  

> > to make the build command depend on the timestamp (which is fairly easy but messy) or  

> I

> > have to put it in the emitter, but I suspect that'll happen *before* (e.g.) libA.so is  

> > rebuilt.

> >

> > ----- Original Message -----

> > From: bill at baddogconsulting.com

> > To: Tom Tanner (BLOOMBERG/ LONDON), scons-users at scons.org

> > At: Mar 17 2014 18:07:45

> >

> > Tom,

> >

> >

> > On March 17, 2014 at 1:19:49 AM, Tom Tanner (BLOOMBERG/ LONDON) (ttanner2 at bloomberg.net)  

> > wrote:

> > > One of our tools has a very strange build avoidance system (which sadly can't be turned  

> > > off).

> >

> > So this build tool is run by SCons?

> >

> > >

> > > It checks the timestamp of the file you feed it and if it is different to the timestamp  

> > of

> > > the target file, it rebuilds the target file and applies the timestamp of the source  

> > file.

> > >

> > > As it runs on executables and shared libraries, it also recurses down all the shared  

> > libraries

> > > referenced.

> > >

> > > I think this is causing moderately strange behaviour in our builds occasionally -  

> if

> > > a library is rebuilt but doesn't change (apart from the timestamp), then the tool is  

> > not

> > > run on it. However, if some other library or executable that depends on that library  

> > is

> > > rebuilt and does change, then the tool is run, and it detects the first library has changed,  

> > > and then runs itself on that library.

> >

> > Can you explain how this is different than how SCons works?

> > If a source down the tree changes, it’s going to rebuild the dependency tree..

> >

> > >

> > > Given that peculiar behaviour, what I think I need to do is to make the target also depend  

> > > on the timestamp of the source. I'm worried about the best way of doing this though.  

> > >

> > > I can't find out the timestamp of the source library until it is built. So

> > > 1) Can I depend on the source files existing, in which case I can make a Value object (we're  

> > > using md5sums exclusively)

> > > 2) If not

> > > a) Can I add a dependency in the generator (i.e. ditto)

> > > b) Should I add an action whose text depends on the timestamp of the source file, using  

> > > something like 'echo ${ function that returns source timestamp }’

> >

> >

> > -Bill

> >

> > _______________________________________________

> > Scons-users mailing list

> > Scons-users at scons.org

> > http://four.pairlist.net/mailman/listinfo/scons-users

> >

>  

>  

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

>  



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://four.pairlist.net/pipermail/scons-users/attachments/20140319/66b14ff0/attachment-0001.html