[Scons-users] Bug causing corruption of .sconsign.dblite

Sun Oct 19 15:21:14 EDT 2014

Piotr,

I applied the patch to the latest SCons source, and it broke several
automated tests.  I do not know whether those tests simply need to be
updated or additional modifications are required.

I am currently working on another issue regarding cross-language scanner
support, and I cannot dedicate the time to investigate at present.  If you
are eager to have this fixed in the next SCons update, consider forking a
repository @ https://bitbucket.org/scons/scons and creating a pull
request.  Additional developer notes can be found @
http://www.scons.org/wiki and there are individuals who can help if you
have additional questions.

V/R,
William

On Fri, Oct 17, 2014 at 8:37 PM, William Blevins <wblevins001 at gmail.com>
wrote:

> Piotr,
>
> A software failure indeed.  After your test steps, the md5sum for beta.h
> is incorrect.  If I update the timestamp for beta.h, main.cpp will
> incorrectly rebuild.
>
> Thank you for a streamlined test case. I made a issue tracker for this
> defect: http://scons.tigris.org/issues/show_bug.cgi?id=2980
>
> V/R,
> William
>
> On Thu, Oct 16, 2014 at 7:58 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
>
>> Piotr,
>>
>> Thanks for supplying the additional information. I will make my best
>> effort to take a close look at the information this weekend.  In the
>> unlikely event that a SCons developer hasn't gotten back to you by end of
>> next week, please ping us with a gentle reminder.
>>
>> V/R,
>> William
>>
>> On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik <
>> piotr.bartosik at gmail.com> wrote:
>>
>>> William,
>>>
>>> please find attached a simple test that reproduces the bug. I tested it
>>> on Linux. Below is my example output with discrepancies highlighted:
>>>
>>> $ ./test.sh ~/tools/scons
>>> Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
>>>
>>> You are in 'detached HEAD' state. You can look around, make experimental
>>> changes and commit them, and you can discard any commits you make in this
>>> state without impacting any branches by performing another checkout.
>>>
>>> If you want to create a new branch to retain commits you create, you may
>>> do so (now or later) by using -b with the checkout command again.
>>> Example:
>>>
>>>   git checkout -b new_branch_name
>>>
>>> HEAD is now at 6ee27c0... initial
>>> scons: Reading SConscript files ...
>>> scons: done reading SConscript files.
>>> scons: Building targets ...
>>> g++ -o main.o -c main.cpp
>>> g++ -o main main.o
>>> scons: done building targets.
>>> Previous HEAD position was 6ee27c0... initial
>>> Switched to branch 'master'
>>> scons: Reading SConscript files ...
>>> scons: done reading SConscript files.
>>> scons: Building targets ...
>>> g++ -o main.o -c main.cpp
>>> scons: done building targets.
>>> === .:
>>> SConstruct: None
>>> alpha.h: 3518d8c3310bba3c2a0da5489f569e14
>>> beta.h: 2ff783593a2224d0574181661ab5f1b7
>>> gamma.h: 2ff783593a2224d0574181661ab5f1b7
>>> main: b53741b91f13474487bbbc1efe5e328b
>>> main.cpp: ebf971b7da9c98f91067a9c601e96259
>>> main.o: faaa740a3704b2c8f034bafd7c1722ff
>>> === /usr/bin:
>>> g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
>>> 3518d8c3310bba3c2a0da5489f569e14  alpha.h
>>> bca4755df4cd2eef7d51bfb007c36dd1  beta.h
>>> 2ff783593a2224d0574181661ab5f1b7  gamma.h
>>> ebf971b7da9c98f91067a9c601e96259  main.cpp
>>>
>>>
>>> Regards,
>>> Piotr
>>>
>>>
>>> 2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>>
>>>> Piotr,
>>>>
>>>> That should be fine.  Try to keep the dataset as simple and lightweight
>>>> as possible, since a test case needs to be created from it.  If you only
>>>> need a single c-file with a few header includes to reproduce, then that
>>>> should be plenty.
>>>>
>>>> If you feel comfortable request a patch addition, you can do so here:
>>>> https://bitbucket.org/scons/scons.
>>>>
>>>> V/R,
>>>> William
>>>>
>>>> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik <
>>>> piotr.bartosik at gmail.com> wrote:
>>>>
>>>>> William,
>>>>>
>>>>> thank you for your answer. I'm stating to prepare a test data set. I'm
>>>>> going to send you a compressed directory versioned with git (with .git dir
>>>>> inside) and an exact list of steps to reproduce the problem. I hope that
>>>>> this format is convenient to you, if not please let me know.
>>>>>
>>>>> Best regards,
>>>>> Piotr
>>>>>
>>>>>
>>>>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>>>>
>>>>>> Piotr,
>>>>>>
>>>>>> Also, as a side question, do you have a set of test case values we
>>>>>> can use for this case?
>>>>>>
>>>>>> V/R,
>>>>>> William
>>>>>>
>>>>>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins <
>>>>>> wblevins001 at gmail.com> wrote:
>>>>>>
>>>>>>> Piotr,
>>>>>>>
>>>>>>> I am sure you already did your homework, and we appreciate your
>>>>>>> input.  Before we start looking at this closely, can you guarantee that the
>>>>>>> following files were not accidentally committed your GIT repository: object
>>>>>>> files, libraries, executables, the .sconsign.dblite file itself?  I
>>>>>>> have seen cases in the past were files like these get
>>>>>>> committed accidentally, and that might cause effects like those you
>>>>>>> described.
>>>>>>>
>>>>>>> V/R,
>>>>>>> William
>>>>>>>
>>>>>>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik <
>>>>>>> piotr.bartosik at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> while using SCons (version 2.3.0) together with git I encountered
>>>>>>>> some problems with MD5 calculations. After rebasing to different revisions
>>>>>>>> of my repository I noticed strange cache misses (the proper .o files should
>>>>>>>> be already in the cache while they were actually rebuilt). This led me to
>>>>>>>> the conclusion that some MD5 values may be calculated incorrectly. In fact,
>>>>>>>> after using the script sconsign.py it became apparent that some MD5s in
>>>>>>>> .sconsign.dblite were incorrect (the sums calculated directly using hashlib
>>>>>>>> were different).
>>>>>>>>
>>>>>>>> To sum up, after this simple procedure:
>>>>>>>>
>>>>>>>> rm .sconsign.dblite
>>>>>>>> git checkout SOME_VERSION_1
>>>>>>>> scons
>>>>>>>> git checkout SOME_VERSION_2
>>>>>>>> scons
>>>>>>>>
>>>>>>>> sconsign.py reported incorrect MD5 sums.
>>>>>>>>
>>>>>>>> As it turned out, the problem is localized in Node/__init__.py. The
>>>>>>>> code
>>>>>>>>
>>>>>>>> then.extend([None] * diff)
>>>>>>>>
>>>>>>>> in line 1050 causes desynchronization between two lists: the
>>>>>>>> current dependencies and info nodes for dependencies from the previous
>>>>>>>> build (containing, among others, previously calculated MD5 sums).
>>>>>>>>
>>>>>>>> For example, consider that in SOME_VERSION_1 a file has five
>>>>>>>> dependencies:
>>>>>>>> A B C D E
>>>>>>>> and in SOME_VERSION_2 a new dependency appears:
>>>>>>>> A B C X D E
>>>>>>>>
>>>>>>>> After line 1050, the lists "then" and "children" look like below:
>>>>>>>> then:     A B C D E None
>>>>>>>> children: A B C X D E
>>>>>>>>
>>>>>>>> Code further below updates children information. As it appears,
>>>>>>>> such desynchronization does not cause any problems if we are checking MD5s
>>>>>>>> each time: they just will be recalculated for X, D and E. But if we also
>>>>>>>> look at the timestamps, the condition of timestamp remaining the same
>>>>>>>> allows SCons to copy the MD5 sum from the "then" array to the current child
>>>>>>>> object. Consequently, if D and E have the same modification date, E's
>>>>>>>> checksum will be incorrectly copied into D's info. And this is exactly the
>>>>>>>> error that I observe.
>>>>>>>>
>>>>>>>> My quick fix builds "then" list in a proper way, .i.e. without
>>>>>>>> desynchronization (please see the attachment):
>>>>>>>> then:     A B C None D E
>>>>>>>> children: A B C X    D E
>>>>>>>>
>>>>>>>> Please look at it and tell me what you think. I don't know the
>>>>>>>> SCons' sources very well so my fix may be a little naive but this is what I
>>>>>>>> can provide for a start.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Piotr
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141019/c5ac0d95/attachment-0001.html>