[Scons-users] Bug causing corruption of .sconsign.dblite

Thu Oct 16 19:58:12 EDT 2014

Piotr,

Thanks for supplying the additional information. I will make my best effort
to take a close look at the information this weekend.  In the unlikely
event that a SCons developer hasn't gotten back to you by end of next week,
please ping us with a gentle reminder.

V/R,
William

On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik <piotr.bartosik at gmail.com>
wrote:

> William,
>
> please find attached a simple test that reproduces the bug. I tested it on
> Linux. Below is my example output with discrepancies highlighted:
>
> $ ./test.sh ~/tools/scons
> Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
>
> You are in 'detached HEAD' state. You can look around, make experimental
> changes and commit them, and you can discard any commits you make in this
> state without impacting any branches by performing another checkout.
>
> If you want to create a new branch to retain commits you create, you may
> do so (now or later) by using -b with the checkout command again. Example:
>
>   git checkout -b new_branch_name
>
> HEAD is now at 6ee27c0... initial
> scons: Reading SConscript files ...
> scons: done reading SConscript files.
> scons: Building targets ...
> g++ -o main.o -c main.cpp
> g++ -o main main.o
> scons: done building targets.
> Previous HEAD position was 6ee27c0... initial
> Switched to branch 'master'
> scons: Reading SConscript files ...
> scons: done reading SConscript files.
> scons: Building targets ...
> g++ -o main.o -c main.cpp
> scons: done building targets.
> === .:
> SConstruct: None
> alpha.h: 3518d8c3310bba3c2a0da5489f569e14
> beta.h: 2ff783593a2224d0574181661ab5f1b7
> gamma.h: 2ff783593a2224d0574181661ab5f1b7
> main: b53741b91f13474487bbbc1efe5e328b
> main.cpp: ebf971b7da9c98f91067a9c601e96259
> main.o: faaa740a3704b2c8f034bafd7c1722ff
> === /usr/bin:
> g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
> 3518d8c3310bba3c2a0da5489f569e14  alpha.h
> bca4755df4cd2eef7d51bfb007c36dd1  beta.h
> 2ff783593a2224d0574181661ab5f1b7  gamma.h
> ebf971b7da9c98f91067a9c601e96259  main.cpp
>
>
> Regards,
> Piotr
>
>
> 2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>
>> Piotr,
>>
>> That should be fine.  Try to keep the dataset as simple and lightweight
>> as possible, since a test case needs to be created from it.  If you only
>> need a single c-file with a few header includes to reproduce, then that
>> should be plenty.
>>
>> If you feel comfortable request a patch addition, you can do so here:
>> https://bitbucket.org/scons/scons.
>>
>> V/R,
>> William
>>
>> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik <piotr.bartosik at gmail.com
>> > wrote:
>>
>>> William,
>>>
>>> thank you for your answer. I'm stating to prepare a test data set. I'm
>>> going to send you a compressed directory versioned with git (with .git dir
>>> inside) and an exact list of steps to reproduce the problem. I hope that
>>> this format is convenient to you, if not please let me know.
>>>
>>> Best regards,
>>> Piotr
>>>
>>>
>>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>>
>>>> Piotr,
>>>>
>>>> Also, as a side question, do you have a set of test case values we can
>>>> use for this case?
>>>>
>>>> V/R,
>>>> William
>>>>
>>>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins <wblevins001 at gmail.com
>>>> > wrote:
>>>>
>>>>> Piotr,
>>>>>
>>>>> I am sure you already did your homework, and we appreciate your
>>>>> input.  Before we start looking at this closely, can you guarantee that the
>>>>> following files were not accidentally committed your GIT repository: object
>>>>> files, libraries, executables, the .sconsign.dblite file itself?  I
>>>>> have seen cases in the past were files like these get
>>>>> committed accidentally, and that might cause effects like those you
>>>>> described.
>>>>>
>>>>> V/R,
>>>>> William
>>>>>
>>>>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik <
>>>>> piotr.bartosik at gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> while using SCons (version 2.3.0) together with git I encountered
>>>>>> some problems with MD5 calculations. After rebasing to different revisions
>>>>>> of my repository I noticed strange cache misses (the proper .o files should
>>>>>> be already in the cache while they were actually rebuilt). This led me to
>>>>>> the conclusion that some MD5 values may be calculated incorrectly. In fact,
>>>>>> after using the script sconsign.py it became apparent that some MD5s in
>>>>>> .sconsign.dblite were incorrect (the sums calculated directly using hashlib
>>>>>> were different).
>>>>>>
>>>>>> To sum up, after this simple procedure:
>>>>>>
>>>>>> rm .sconsign.dblite
>>>>>> git checkout SOME_VERSION_1
>>>>>> scons
>>>>>> git checkout SOME_VERSION_2
>>>>>> scons
>>>>>>
>>>>>> sconsign.py reported incorrect MD5 sums.
>>>>>>
>>>>>> As it turned out, the problem is localized in Node/__init__.py. The
>>>>>> code
>>>>>>
>>>>>> then.extend([None] * diff)
>>>>>>
>>>>>> in line 1050 causes desynchronization between two lists: the current
>>>>>> dependencies and info nodes for dependencies from the previous build
>>>>>> (containing, among others, previously calculated MD5 sums).
>>>>>>
>>>>>> For example, consider that in SOME_VERSION_1 a file has five
>>>>>> dependencies:
>>>>>> A B C D E
>>>>>> and in SOME_VERSION_2 a new dependency appears:
>>>>>> A B C X D E
>>>>>>
>>>>>> After line 1050, the lists "then" and "children" look like below:
>>>>>> then:     A B C D E None
>>>>>> children: A B C X D E
>>>>>>
>>>>>> Code further below updates children information. As it appears, such
>>>>>> desynchronization does not cause any problems if we are checking MD5s each
>>>>>> time: they just will be recalculated for X, D and E. But if we also look at
>>>>>> the timestamps, the condition of timestamp remaining the same allows SCons
>>>>>> to copy the MD5 sum from the "then" array to the current child object.
>>>>>> Consequently, if D and E have the same modification date, E's checksum will
>>>>>> be incorrectly copied into D's info. And this is exactly the error that I
>>>>>> observe.
>>>>>>
>>>>>> My quick fix builds "then" list in a proper way, .i.e. without
>>>>>> desynchronization (please see the attachment):
>>>>>> then:     A B C None D E
>>>>>> children: A B C X    D E
>>>>>>
>>>>>> Please look at it and tell me what you think. I don't know the SCons'
>>>>>> sources very well so my fix may be a little naive but this is what I can
>>>>>> provide for a start.
>>>>>>
>>>>>> Thanks,
>>>>>> Piotr
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141016/9e859179/attachment.html>