[Scons-users] Bug causing corruption of .sconsign.dblite
Piotr Bartosik
piotr.bartosik at gmail.com
Thu Oct 16 10:40:40 EDT 2014
William,
please find attached a simple test that reproduces the bug. I tested it on
Linux. Below is my example output with discrepancies highlighted:
$ ./test.sh ~/tools/scons
Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at 6ee27c0... initial
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o main.o -c main.cpp
g++ -o main main.o
scons: done building targets.
Previous HEAD position was 6ee27c0... initial
Switched to branch 'master'
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o main.o -c main.cpp
scons: done building targets.
=== .:
SConstruct: None
alpha.h: 3518d8c3310bba3c2a0da5489f569e14
beta.h: 2ff783593a2224d0574181661ab5f1b7
gamma.h: 2ff783593a2224d0574181661ab5f1b7
main: b53741b91f13474487bbbc1efe5e328b
main.cpp: ebf971b7da9c98f91067a9c601e96259
main.o: faaa740a3704b2c8f034bafd7c1722ff
=== /usr/bin:
g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
3518d8c3310bba3c2a0da5489f569e14 alpha.h
bca4755df4cd2eef7d51bfb007c36dd1 beta.h
2ff783593a2224d0574181661ab5f1b7 gamma.h
ebf971b7da9c98f91067a9c601e96259 main.cpp
Regards,
Piotr
2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
> Piotr,
>
> That should be fine. Try to keep the dataset as simple and lightweight as
> possible, since a test case needs to be created from it. If you only need
> a single c-file with a few header includes to reproduce, then that should
> be plenty.
>
> If you feel comfortable request a patch addition, you can do so here:
> https://bitbucket.org/scons/scons.
>
> V/R,
> William
>
> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik <piotr.bartosik at gmail.com>
> wrote:
>
>> William,
>>
>> thank you for your answer. I'm stating to prepare a test data set. I'm
>> going to send you a compressed directory versioned with git (with .git dir
>> inside) and an exact list of steps to reproduce the problem. I hope that
>> this format is convenient to you, if not please let me know.
>>
>> Best regards,
>> Piotr
>>
>>
>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>
>>> Piotr,
>>>
>>> Also, as a side question, do you have a set of test case values we can
>>> use for this case?
>>>
>>> V/R,
>>> William
>>>
>>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>>> Piotr,
>>>>
>>>> I am sure you already did your homework, and we appreciate your input.
>>>> Before we start looking at this closely, can you guarantee that the
>>>> following files were not accidentally committed your GIT repository: object
>>>> files, libraries, executables, the .sconsign.dblite file itself? I
>>>> have seen cases in the past were files like these get
>>>> committed accidentally, and that might cause effects like those you
>>>> described.
>>>>
>>>> V/R,
>>>> William
>>>>
>>>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik <
>>>> piotr.bartosik at gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> while using SCons (version 2.3.0) together with git I encountered some
>>>>> problems with MD5 calculations. After rebasing to different revisions of my
>>>>> repository I noticed strange cache misses (the proper .o files should be
>>>>> already in the cache while they were actually rebuilt). This led me to the
>>>>> conclusion that some MD5 values may be calculated incorrectly. In fact,
>>>>> after using the script sconsign.py it became apparent that some MD5s in
>>>>> .sconsign.dblite were incorrect (the sums calculated directly using hashlib
>>>>> were different).
>>>>>
>>>>> To sum up, after this simple procedure:
>>>>>
>>>>> rm .sconsign.dblite
>>>>> git checkout SOME_VERSION_1
>>>>> scons
>>>>> git checkout SOME_VERSION_2
>>>>> scons
>>>>>
>>>>> sconsign.py reported incorrect MD5 sums.
>>>>>
>>>>> As it turned out, the problem is localized in Node/__init__.py. The
>>>>> code
>>>>>
>>>>> then.extend([None] * diff)
>>>>>
>>>>> in line 1050 causes desynchronization between two lists: the current
>>>>> dependencies and info nodes for dependencies from the previous build
>>>>> (containing, among others, previously calculated MD5 sums).
>>>>>
>>>>> For example, consider that in SOME_VERSION_1 a file has five
>>>>> dependencies:
>>>>> A B C D E
>>>>> and in SOME_VERSION_2 a new dependency appears:
>>>>> A B C X D E
>>>>>
>>>>> After line 1050, the lists "then" and "children" look like below:
>>>>> then: A B C D E None
>>>>> children: A B C X D E
>>>>>
>>>>> Code further below updates children information. As it appears, such
>>>>> desynchronization does not cause any problems if we are checking MD5s each
>>>>> time: they just will be recalculated for X, D and E. But if we also look at
>>>>> the timestamps, the condition of timestamp remaining the same allows SCons
>>>>> to copy the MD5 sum from the "then" array to the current child object.
>>>>> Consequently, if D and E have the same modification date, E's checksum will
>>>>> be incorrectly copied into D's info. And this is exactly the error that I
>>>>> observe.
>>>>>
>>>>> My quick fix builds "then" list in a proper way, .i.e. without
>>>>> desynchronization (please see the attachment):
>>>>> then: A B C None D E
>>>>> children: A B C X D E
>>>>>
>>>>> Please look at it and tell me what you think. I don't know the SCons'
>>>>> sources very well so my fix may be a little naive but this is what I can
>>>>> provide for a start.
>>>>>
>>>>> Thanks,
>>>>> Piotr
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141016/70d1436b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: csig_bug.tar.gz
Type: application/x-gzip
Size: 11110 bytes
Desc: not available
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141016/70d1436b/attachment-0001.bin>
More information about the Scons-users
mailing list