[Scons-users] Bug causing corruption of .sconsign.dblite
Andrew C. Morrow
andrew.c.morrow at gmail.com
Mon Aug 24 12:48:16 EDT 2015
I just was bitten by this bug during a bisect. The MD5-Timestamp
decider is very useful for iterative developer builds so I usually
have it enabled. Similar for CacheDir, especially during a bisect
since a particular source file version may be revisited several times.
It is unfortunate that these features interact poorly.
Is there any chance of getting this issue on the roadmap to be fixed
in SCons 2.4?
Thanks,
Andrew
On Fri, Oct 24, 2014 at 6:47 PM, William Blevins <wblevins001 at gmail.com> wrote:
> Andrew,
>
> Once there is a fix, then a release is inevitable. There isn't an expected
> completion date.
>
> I cannot speak for everyone else, but I feel that this is a lower priority
> issue, so I will not investigate further until I have finished
> cross-language scanner enhancements. If you would like this resolved ASAP,
> please feel free to contribute a finalized patch with tests.
>
> V/R,
> William
>
> On Fri, Oct 24, 2014 at 10:21 AM, Andrew C. Morrow
> <andrew.c.morrow at gmail.com> wrote:
>>
>>
>> Hi -
>>
>> Is there a possibility for a 2.3.5 release containing a fix for this, once
>> one is completed?
>>
>> Thanks,
>> Andrew
>>
>>
>> On Mon, Oct 20, 2014 at 10:20 AM, William Blevins <wblevins001 at gmail.com>
>> wrote:
>>>
>>> Stijn,
>>>
>>> Negative. The issue has nothing to do with GIT.
>>>
>>> The bug exists within the "MD5-timestamp" decider. This decider can
>>> falsely mark Nodes as "not up-to-date" because of how the dependency cache
>>> is expanded when new dependencies are added to a Node. In worst case,
>>> sometimes code will get compiled that wasn't necessary, but only after the
>>> timestamp on those files would change; thus, it's a performance bug, but in
>>> most cases will still be faster than using the "MD5" decider since it
>>> doesn't need to compute the MD5 sums every pass.
>>>
>>> V/R,
>>> William
>>>
>>> On Mon, Oct 20, 2014 at 4:31 AM, Stijn De Ruyck
>>> <Stijn.DeRuyck at onsemi.com> wrote:
>>>>
>>>> I’m trying to understand this issue. Can someone summarize?
>>>>
>>>>
>>>>
>>>> Does it mean Scons 2.3.0 is essentially unusable when combined with git?
>>>>
>>>> I’m currently converting 2 software projects to Scons, but I’m stuck
>>>> with 2.3.0 because of Python dependencies…
>>>>
>>>>
>>>>
>>>> Any checkout, pull, rebase, reset, … can corrupt the .sconsign.dblite
>>>> file?
>>>>
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Stijn
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of
>>>> William Blevins
>>>> Sent: Sunday, October 19, 2014 9:21 PM
>>>> To: SCons users mailing list
>>>> Subject: Re: [Scons-users] Bug causing corruption of .sconsign.dblite
>>>>
>>>>
>>>>
>>>> Piotr,
>>>>
>>>>
>>>>
>>>> I applied the patch to the latest SCons source, and it broke several
>>>> automated tests. I do not know whether those tests simply need to be
>>>> updated or additional modifications are required.
>>>>
>>>>
>>>>
>>>> I am currently working on another issue regarding cross-language scanner
>>>> support, and I cannot dedicate the time to investigate at present. If you
>>>> are eager to have this fixed in the next SCons update, consider forking a
>>>> repository @ https://bitbucket.org/scons/scons and creating a pull request.
>>>> Additional developer notes can be found @ http://www.scons.org/wiki and
>>>> there are individuals who can help if you have additional questions.
>>>>
>>>>
>>>>
>>>> V/R,
>>>>
>>>> William
>>>>
>>>>
>>>>
>>>> On Fri, Oct 17, 2014 at 8:37 PM, William Blevins <wblevins001 at gmail.com>
>>>> wrote:
>>>>
>>>> Piotr,
>>>>
>>>>
>>>>
>>>> A software failure indeed. After your test steps, the md5sum for beta.h
>>>> is incorrect. If I update the timestamp for beta.h, main.cpp will
>>>> incorrectly rebuild.
>>>>
>>>>
>>>>
>>>> Thank you for a streamlined test case. I made a issue tracker for this
>>>> defect: http://scons.tigris.org/issues/show_bug.cgi?id=2980
>>>>
>>>>
>>>>
>>>> V/R,
>>>>
>>>> William
>>>>
>>>>
>>>>
>>>> On Thu, Oct 16, 2014 at 7:58 PM, William Blevins <wblevins001 at gmail.com>
>>>> wrote:
>>>>
>>>> Piotr,
>>>>
>>>>
>>>>
>>>> Thanks for supplying the additional information. I will make my best
>>>> effort to take a close look at the information this weekend. In the
>>>> unlikely event that a SCons developer hasn't gotten back to you by end of
>>>> next week, please ping us with a gentle reminder.
>>>>
>>>>
>>>>
>>>> V/R,
>>>>
>>>> William
>>>>
>>>>
>>>>
>>>> On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik
>>>> <piotr.bartosik at gmail.com> wrote:
>>>>
>>>> William,
>>>>
>>>> please find attached a simple test that reproduces the bug. I tested it
>>>> on Linux. Below is my example output with discrepancies highlighted:
>>>>
>>>> $ ./test.sh ~/tools/scons
>>>> Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
>>>>
>>>> You are in 'detached HEAD' state. You can look around, make experimental
>>>> changes and commit them, and you can discard any commits you make in
>>>> this
>>>> state without impacting any branches by performing another checkout.
>>>>
>>>> If you want to create a new branch to retain commits you create, you may
>>>> do so (now or later) by using -b with the checkout command again.
>>>> Example:
>>>>
>>>> git checkout -b new_branch_name
>>>>
>>>> HEAD is now at 6ee27c0... initial
>>>> scons: Reading SConscript files ...
>>>> scons: done reading SConscript files.
>>>> scons: Building targets ...
>>>> g++ -o main.o -c main.cpp
>>>> g++ -o main main.o
>>>> scons: done building targets.
>>>> Previous HEAD position was 6ee27c0... initial
>>>> Switched to branch 'master'
>>>> scons: Reading SConscript files ...
>>>> scons: done reading SConscript files.
>>>> scons: Building targets ...
>>>> g++ -o main.o -c main.cpp
>>>> scons: done building targets.
>>>> === .:
>>>> SConstruct: None
>>>> alpha.h: 3518d8c3310bba3c2a0da5489f569e14
>>>> beta.h: 2ff783593a2224d0574181661ab5f1b7
>>>> gamma.h: 2ff783593a2224d0574181661ab5f1b7
>>>> main: b53741b91f13474487bbbc1efe5e328b
>>>> main.cpp: ebf971b7da9c98f91067a9c601e96259
>>>> main.o: faaa740a3704b2c8f034bafd7c1722ff
>>>> === /usr/bin:
>>>> g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
>>>> 3518d8c3310bba3c2a0da5489f569e14 alpha.h
>>>> bca4755df4cd2eef7d51bfb007c36dd1 beta.h
>>>> 2ff783593a2224d0574181661ab5f1b7 gamma.h
>>>> ebf971b7da9c98f91067a9c601e96259 main.cpp
>>>>
>>>> Regards,
>>>>
>>>> Piotr
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>>>
>>>> Piotr,
>>>>
>>>>
>>>>
>>>> That should be fine. Try to keep the dataset as simple and lightweight
>>>> as possible, since a test case needs to be created from it. If you only
>>>> need a single c-file with a few header includes to reproduce, then that
>>>> should be plenty.
>>>>
>>>>
>>>>
>>>> If you feel comfortable request a patch addition, you can do so here:
>>>> https://bitbucket.org/scons/scons.
>>>>
>>>>
>>>>
>>>> V/R,
>>>>
>>>> William
>>>>
>>>>
>>>>
>>>> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik
>>>> <piotr.bartosik at gmail.com> wrote:
>>>>
>>>> William,
>>>>
>>>> thank you for your answer. I'm stating to prepare a test data set. I'm
>>>> going to send you a compressed directory versioned with git (with .git dir
>>>> inside) and an exact list of steps to reproduce the problem. I hope that
>>>> this format is convenient to you, if not please let me know.
>>>>
>>>> Best regards,
>>>> Piotr
>>>>
>>>>
>>>>
>>>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>>>
>>>> Piotr,
>>>>
>>>>
>>>>
>>>> Also, as a side question, do you have a set of test case values we can
>>>> use for this case?
>>>>
>>>>
>>>>
>>>> V/R,
>>>>
>>>> William
>>>>
>>>>
>>>>
>>>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins <wblevins001 at gmail.com>
>>>> wrote:
>>>>
>>>> Piotr,
>>>>
>>>>
>>>>
>>>> I am sure you already did your homework, and we appreciate your input.
>>>> Before we start looking at this closely, can you guarantee that the
>>>> following files were not accidentally committed your GIT repository: object
>>>> files, libraries, executables, the .sconsign.dblite file itself? I have
>>>> seen cases in the past were files like these get committed accidentally, and
>>>> that might cause effects like those you described.
>>>>
>>>>
>>>>
>>>> V/R,
>>>>
>>>> William
>>>>
>>>>
>>>>
>>>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik
>>>> <piotr.bartosik at gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> while using SCons (version 2.3.0) together with git I encountered some
>>>> problems with MD5 calculations. After rebasing to different revisions of my
>>>> repository I noticed strange cache misses (the proper .o files should be
>>>> already in the cache while they were actually rebuilt). This led me to the
>>>> conclusion that some MD5 values may be calculated incorrectly. In fact,
>>>> after using the script sconsign.py it became apparent that some MD5s in
>>>> .sconsign.dblite were incorrect (the sums calculated directly using hashlib
>>>> were different).
>>>>
>>>> To sum up, after this simple procedure:
>>>>
>>>> rm .sconsign.dblite
>>>>
>>>> git checkout SOME_VERSION_1
>>>>
>>>> scons
>>>>
>>>> git checkout SOME_VERSION_2
>>>>
>>>> scons
>>>>
>>>> sconsign.py reported incorrect MD5 sums.
>>>>
>>>> As it turned out, the problem is localized in Node/__init__.py. The code
>>>>
>>>> then.extend([None] * diff)
>>>>
>>>> in line 1050 causes desynchronization between two lists: the current
>>>> dependencies and info nodes for dependencies from the previous build
>>>> (containing, among others, previously calculated MD5 sums).
>>>>
>>>>
>>>>
>>>> For example, consider that in SOME_VERSION_1 a file has five
>>>> dependencies:
>>>>
>>>> A B C D E
>>>>
>>>> and in SOME_VERSION_2 a new dependency appears:
>>>>
>>>> A B C X D E
>>>>
>>>> After line 1050, the lists "then" and "children" look like below:
>>>>
>>>> then: A B C D E None
>>>>
>>>> children: A B C X D E
>>>>
>>>> Code further below updates children information. As it appears, such
>>>> desynchronization does not cause any problems if we are checking MD5s each
>>>> time: they just will be recalculated for X, D and E. But if we also look at
>>>> the timestamps, the condition of timestamp remaining the same allows SCons
>>>> to copy the MD5 sum from the "then" array to the current child object.
>>>> Consequently, if D and E have the same modification date, E's checksum will
>>>> be incorrectly copied into D's info. And this is exactly the error that I
>>>> observe.
>>>>
>>>> My quick fix builds "then" list in a proper way, .i.e. without
>>>> desynchronization (please see the attachment):
>>>>
>>>> then: A B C None D E
>>>>
>>>> children: A B C X D E
>>>>
>>>> Please look at it and tell me what you think. I don't know the SCons'
>>>> sources very well so my fix may be a little naive but this is what I can
>>>> provide for a start.
>>>>
>>>> Thanks,
>>>>
>>>> Piotr
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
More information about the Scons-users
mailing list