[Scons-users] Bug causing corruption of .sconsign.dblite

Mon Oct 20 04:31:18 EDT 2014

I’m trying to understand this issue. Can someone summarize?

Does it mean Scons 2.3.0 is essentially unusable when combined with git?
I’m currently converting 2 software projects to Scons, but I’m stuck with 2.3.0 because of Python dependencies…

Any checkout, pull, rebase, reset, … can corrupt the .sconsign.dblite file?

Best regards,
Stijn

From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of William Blevins
Sent: Sunday, October 19, 2014 9:21 PM
To: SCons users mailing list
Subject: Re: [Scons-users] Bug causing corruption of .sconsign.dblite

Piotr,

I applied the patch to the latest SCons source, and it broke several automated tests.  I do not know whether those tests simply need to be updated or additional modifications are required.

I am currently working on another issue regarding cross-language scanner support, and I cannot dedicate the time to investigate at present.  If you are eager to have this fixed in the next SCons update, consider forking a repository @ https://bitbucket.org/scons/scons and creating a pull request.  Additional developer notes can be found @ http://www.scons.org/wiki and there are individuals who can help if you have additional questions.

V/R,
William

On Fri, Oct 17, 2014 at 8:37 PM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:
Piotr,

A software failure indeed.  After your test steps, the md5sum for beta.h is incorrect.  If I update the timestamp for beta.h, main.cpp will incorrectly rebuild.

Thank you for a streamlined test case. I made a issue tracker for this defect: http://scons.tigris.org/issues/show_bug.cgi?id=2980

V/R,
William

On Thu, Oct 16, 2014 at 7:58 PM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:
Piotr,

Thanks for supplying the additional information. I will make my best effort to take a close look at the information this weekend.  In the unlikely event that a SCons developer hasn't gotten back to you by end of next week, please ping us with a gentle reminder.

V/R,
William

On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik <piotr.bartosik at gmail.com<mailto:piotr.bartosik at gmail.com>> wrote:
William,
please find attached a simple test that reproduces the bug. I tested it on Linux. Below is my example output with discrepancies highlighted:

$ ./test.sh ~/tools/scons
Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 6ee27c0... initial
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o main.o -c main.cpp
g++ -o main main.o
scons: done building targets.
Previous HEAD position was 6ee27c0... initial
Switched to branch 'master'
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o main.o -c main.cpp
scons: done building targets.
=== .:
SConstruct: None
alpha.h: 3518d8c3310bba3c2a0da5489f569e14
beta.h: 2ff783593a2224d0574181661ab5f1b7
gamma.h: 2ff783593a2224d0574181661ab5f1b7
main: b53741b91f13474487bbbc1efe5e328b
main.cpp: ebf971b7da9c98f91067a9c601e96259
main.o: faaa740a3704b2c8f034bafd7c1722ff
=== /usr/bin:
g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
3518d8c3310bba3c2a0da5489f569e14  alpha.h
bca4755df4cd2eef7d51bfb007c36dd1  beta.h
2ff783593a2224d0574181661ab5f1b7  gamma.h
ebf971b7da9c98f91067a9c601e96259  main.cpp

Regards,
Piotr

2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>>:
Piotr,

That should be fine.  Try to keep the dataset as simple and lightweight as possible, since a test case needs to be created from it.  If you only need a single c-file with a few header includes to reproduce, then that should be plenty.

If you feel comfortable request a patch addition, you can do so here: https://bitbucket.org/scons/scons.

V/R,
William

On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik <piotr.bartosik at gmail.com<mailto:piotr.bartosik at gmail.com>> wrote:
William,
thank you for your answer. I'm stating to prepare a test data set. I'm going to send you a compressed directory versioned with git (with .git dir inside) and an exact list of steps to reproduce the problem. I hope that this format is convenient to you, if not please let me know.
Best regards,
Piotr

2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>>:
Piotr,

Also, as a side question, do you have a set of test case values we can use for this case?

V/R,
William

On Tue, Oct 14, 2014 at 7:23 PM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:
Piotr,

I am sure you already did your homework, and we appreciate your input.  Before we start looking at this closely, can you guarantee that the following files were not accidentally committed your GIT repository: object files, libraries, executables, the .sconsign.dblite file itself?  I have seen cases in the past were files like these get committed accidentally, and that might cause effects like those you described.

V/R,
William

On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik <piotr.bartosik at gmail.com<mailto:piotr.bartosik at gmail.com>> wrote:
Hi,
while using SCons (version 2.3.0) together with git I encountered some problems with MD5 calculations. After rebasing to different revisions of my repository I noticed strange cache misses (the proper .o files should be already in the cache while they were actually rebuilt). This led me to the conclusion that some MD5 values may be calculated incorrectly. In fact, after using the script sconsign.py it became apparent that some MD5s in .sconsign.dblite were incorrect (the sums calculated directly using hashlib were different).
To sum up, after this simple procedure:

rm .sconsign.dblite
git checkout SOME_VERSION_1
scons
git checkout SOME_VERSION_2
scons

sconsign.py reported incorrect MD5 sums.
As it turned out, the problem is localized in Node/__init__.py. The code

then.extend([None] * diff)
in line 1050 causes desynchronization between two lists: the current dependencies and info nodes for dependencies from the previous build (containing, among others, previously calculated MD5 sums).

For example, consider that in SOME_VERSION_1 a file has five dependencies:
A B C D E
and in SOME_VERSION_2 a new dependency appears:
A B C X D E
After line 1050, the lists "then" and "children" look like below:
then:     A B C D E None
children: A B C X D E
Code further below updates children information. As it appears, such desynchronization does not cause any problems if we are checking MD5s each time: they just will be recalculated for X, D and E. But if we also look at the timestamps, the condition of timestamp remaining the same allows SCons to copy the MD5 sum from the "then" array to the current child object. Consequently, if D and E have the same modification date, E's checksum will be incorrectly copied into D's info. And this is exactly the error that I observe.
My quick fix builds "then" list in a proper way, .i.e. without desynchronization (please see the attachment):
then:     A B C None D E
children: A B C X    D E
Please look at it and tell me what you think. I don't know the SCons' sources very well so my fix may be a little naive but this is what I can provide for a start.
Thanks,
Piotr

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141020/96ec4374/attachment-0001.html>