[Scons-users] Bug causing corruption of .sconsign.dblite

Fri Oct 17 20:37:50 EDT 2014

Piotr,

A software failure indeed.  After your test steps, the md5sum for beta.h is
incorrect.  If I update the timestamp for beta.h, main.cpp will incorrectly
rebuild.

Thank you for a streamlined test case. I made a issue tracker for this
defect: http://scons.tigris.org/issues/show_bug.cgi?id=2980

V/R,
William

On Thu, Oct 16, 2014 at 7:58 PM, William Blevins <wblevins001 at gmail.com>
wrote:

> Piotr,
>
> Thanks for supplying the additional information. I will make my best
> effort to take a close look at the information this weekend.  In the
> unlikely event that a SCons developer hasn't gotten back to you by end of
> next week, please ping us with a gentle reminder.
>
> V/R,
> William
>
> On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik <piotr.bartosik at gmail.com
> > wrote:
>
>> William,
>>
>> please find attached a simple test that reproduces the bug. I tested it
>> on Linux. Below is my example output with discrepancies highlighted:
>>
>> $ ./test.sh ~/tools/scons
>> Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
>>
>> You are in 'detached HEAD' state. You can look around, make experimental
>> changes and commit them, and you can discard any commits you make in this
>> state without impacting any branches by performing another checkout.
>>
>> If you want to create a new branch to retain commits you create, you may
>> do so (now or later) by using -b with the checkout command again. Example:
>>
>>   git checkout -b new_branch_name
>>
>> HEAD is now at 6ee27c0... initial
>> scons: Reading SConscript files ...
>> scons: done reading SConscript files.
>> scons: Building targets ...
>> g++ -o main.o -c main.cpp
>> g++ -o main main.o
>> scons: done building targets.
>> Previous HEAD position was 6ee27c0... initial
>> Switched to branch 'master'
>> scons: Reading SConscript files ...
>> scons: done reading SConscript files.
>> scons: Building targets ...
>> g++ -o main.o -c main.cpp
>> scons: done building targets.
>> === .:
>> SConstruct: None
>> alpha.h: 3518d8c3310bba3c2a0da5489f569e14
>> beta.h: 2ff783593a2224d0574181661ab5f1b7
>> gamma.h: 2ff783593a2224d0574181661ab5f1b7
>> main: b53741b91f13474487bbbc1efe5e328b
>> main.cpp: ebf971b7da9c98f91067a9c601e96259
>> main.o: faaa740a3704b2c8f034bafd7c1722ff
>> === /usr/bin:
>> g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
>> 3518d8c3310bba3c2a0da5489f569e14  alpha.h
>> bca4755df4cd2eef7d51bfb007c36dd1  beta.h
>> 2ff783593a2224d0574181661ab5f1b7  gamma.h
>> ebf971b7da9c98f91067a9c601e96259  main.cpp
>>
>>
>> Regards,
>> Piotr
>>
>>
>> 2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>
>>> Piotr,
>>>
>>> That should be fine.  Try to keep the dataset as simple and lightweight
>>> as possible, since a test case needs to be created from it.  If you only
>>> need a single c-file with a few header includes to reproduce, then that
>>> should be plenty.
>>>
>>> If you feel comfortable request a patch addition, you can do so here:
>>> https://bitbucket.org/scons/scons.
>>>
>>> V/R,
>>> William
>>>
>>> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik <
>>> piotr.bartosik at gmail.com> wrote:
>>>
>>>> William,
>>>>
>>>> thank you for your answer. I'm stating to prepare a test data set. I'm
>>>> going to send you a compressed directory versioned with git (with .git dir
>>>> inside) and an exact list of steps to reproduce the problem. I hope that
>>>> this format is convenient to you, if not please let me know.
>>>>
>>>> Best regards,
>>>> Piotr
>>>>
>>>>
>>>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>>>
>>>>> Piotr,
>>>>>
>>>>> Also, as a side question, do you have a set of test case values we can
>>>>> use for this case?
>>>>>
>>>>> V/R,
>>>>> William
>>>>>
>>>>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins <
>>>>> wblevins001 at gmail.com> wrote:
>>>>>
>>>>>> Piotr,
>>>>>>
>>>>>> I am sure you already did your homework, and we appreciate your
>>>>>> input.  Before we start looking at this closely, can you guarantee that the
>>>>>> following files were not accidentally committed your GIT repository: object
>>>>>> files, libraries, executables, the .sconsign.dblite file itself?  I
>>>>>> have seen cases in the past were files like these get
>>>>>> committed accidentally, and that might cause effects like those you
>>>>>> described.
>>>>>>
>>>>>> V/R,
>>>>>> William
>>>>>>
>>>>>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik <
>>>>>> piotr.bartosik at gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> while using SCons (version 2.3.0) together with git I encountered
>>>>>>> some problems with MD5 calculations. After rebasing to different revisions
>>>>>>> of my repository I noticed strange cache misses (the proper .o files should
>>>>>>> be already in the cache while they were actually rebuilt). This led me to
>>>>>>> the conclusion that some MD5 values may be calculated incorrectly. In fact,
>>>>>>> after using the script sconsign.py it became apparent that some MD5s in
>>>>>>> .sconsign.dblite were incorrect (the sums calculated directly using hashlib
>>>>>>> were different).
>>>>>>>
>>>>>>> To sum up, after this simple procedure:
>>>>>>>
>>>>>>> rm .sconsign.dblite
>>>>>>> git checkout SOME_VERSION_1
>>>>>>> scons
>>>>>>> git checkout SOME_VERSION_2
>>>>>>> scons
>>>>>>>
>>>>>>> sconsign.py reported incorrect MD5 sums.
>>>>>>>
>>>>>>> As it turned out, the problem is localized in Node/__init__.py. The
>>>>>>> code
>>>>>>>
>>>>>>> then.extend([None] * diff)
>>>>>>>
>>>>>>> in line 1050 causes desynchronization between two lists: the current
>>>>>>> dependencies and info nodes for dependencies from the previous build
>>>>>>> (containing, among others, previously calculated MD5 sums).
>>>>>>>
>>>>>>> For example, consider that in SOME_VERSION_1 a file has five
>>>>>>> dependencies:
>>>>>>> A B C D E
>>>>>>> and in SOME_VERSION_2 a new dependency appears:
>>>>>>> A B C X D E
>>>>>>>
>>>>>>> After line 1050, the lists "then" and "children" look like below:
>>>>>>> then:     A B C D E None
>>>>>>> children: A B C X D E
>>>>>>>
>>>>>>> Code further below updates children information. As it appears, such
>>>>>>> desynchronization does not cause any problems if we are checking MD5s each
>>>>>>> time: they just will be recalculated for X, D and E. But if we also look at
>>>>>>> the timestamps, the condition of timestamp remaining the same allows SCons
>>>>>>> to copy the MD5 sum from the "then" array to the current child object.
>>>>>>> Consequently, if D and E have the same modification date, E's checksum will
>>>>>>> be incorrectly copied into D's info. And this is exactly the error that I
>>>>>>> observe.
>>>>>>>
>>>>>>> My quick fix builds "then" list in a proper way, .i.e. without
>>>>>>> desynchronization (please see the attachment):
>>>>>>> then:     A B C None D E
>>>>>>> children: A B C X    D E
>>>>>>>
>>>>>>> Please look at it and tell me what you think. I don't know the
>>>>>>> SCons' sources very well so my fix may be a little naive but this is what I
>>>>>>> can provide for a start.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Piotr
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141017/e798daa5/attachment-0001.html>