[Scons-users] Bug causing corruption of .sconsign.dblite

Fri Oct 24 10:21:31 EDT 2014

Hi -

Is there a possibility for a 2.3.5 release containing a fix for this, once
one is completed?

Thanks,
Andrew

On Mon, Oct 20, 2014 at 10:20 AM, William Blevins <wblevins001 at gmail.com>
wrote:

> Stijn,
>
> Negative.  The issue has nothing to do with GIT.
>
> The bug exists within the "MD5-timestamp" decider.  This decider can
> falsely mark Nodes as "not up-to-date" because of how the dependency cache
> is expanded when new dependencies are added to a Node.  In worst case,
> sometimes code will get compiled that wasn't necessary, but only after the
> timestamp on those files would change; thus, it's a performance bug, but in
> most cases will still be faster than using the "MD5" decider since it
> doesn't need to compute the MD5 sums every pass.
>
> V/R,
> William
>
> On Mon, Oct 20, 2014 at 4:31 AM, Stijn De Ruyck <Stijn.DeRuyck at onsemi.com>
> wrote:
>
>>  I’m trying to understand this issue. Can someone summarize?
>>
>>
>>
>> Does it mean Scons 2.3.0 is essentially unusable when combined with git?
>>
>> I’m currently converting 2 software projects to Scons, but I’m stuck with
>> 2.3.0 because of Python dependencies…
>>
>>
>>
>> Any checkout, pull, rebase, reset, … can corrupt the .sconsign.dblite
>> file?
>>
>>
>>
>> Best regards,
>>
>> Stijn
>>
>>
>>
>>
>>
>> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf Of
>> *William Blevins
>> *Sent:* Sunday, October 19, 2014 9:21 PM
>> *To:* SCons users mailing list
>> *Subject:* Re: [Scons-users] Bug causing corruption of .sconsign.dblite
>>
>>
>>
>> Piotr,
>>
>>
>>
>> I applied the patch to the latest SCons source, and it broke several
>> automated tests.  I do not know whether those tests simply need to be
>> updated or additional modifications are required.
>>
>>
>>
>> I am currently working on another issue regarding cross-language scanner
>> support, and I cannot dedicate the time to investigate at present.  If you
>> are eager to have this fixed in the next SCons update, consider forking a
>> repository @ https://bitbucket.org/scons/scons and creating a pull
>> request.  Additional developer notes can be found @
>> http://www.scons.org/wiki and there are individuals who can help if you
>> have additional questions.
>>
>>
>>
>> V/R,
>>
>> William
>>
>>
>>
>> On Fri, Oct 17, 2014 at 8:37 PM, William Blevins <wblevins001 at gmail.com>
>> wrote:
>>
>> Piotr,
>>
>>
>>
>> A software failure indeed.  After your test steps, the md5sum for beta.h
>> is incorrect.  If I update the timestamp for beta.h, main.cpp will
>> incorrectly rebuild.
>>
>>
>>
>> Thank you for a streamlined test case. I made a issue tracker for this
>> defect: http://scons.tigris.org/issues/show_bug.cgi?id=2980
>>
>>
>>
>> V/R,
>>
>> William
>>
>>
>>
>> On Thu, Oct 16, 2014 at 7:58 PM, William Blevins <wblevins001 at gmail.com>
>> wrote:
>>
>> Piotr,
>>
>>
>>
>> Thanks for supplying the additional information. I will make my best
>> effort to take a close look at the information this weekend.  In the
>> unlikely event that a SCons developer hasn't gotten back to you by end of
>> next week, please ping us with a gentle reminder.
>>
>>
>>
>> V/R,
>>
>> William
>>
>>
>>
>> On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik <
>> piotr.bartosik at gmail.com> wrote:
>>
>> William,
>>
>> please find attached a simple test that reproduces the bug. I tested it
>> on Linux. Below is my example output with discrepancies highlighted:
>>
>> $ ./test.sh ~/tools/scons
>> Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
>>
>> You are in 'detached HEAD' state. You can look around, make experimental
>> changes and commit them, and you can discard any commits you make in this
>> state without impacting any branches by performing another checkout.
>>
>> If you want to create a new branch to retain commits you create, you may
>> do so (now or later) by using -b with the checkout command again. Example:
>>
>>   git checkout -b new_branch_name
>>
>> HEAD is now at 6ee27c0... initial
>> scons: Reading SConscript files ...
>> scons: done reading SConscript files.
>> scons: Building targets ...
>> g++ -o main.o -c main.cpp
>> g++ -o main main.o
>> scons: done building targets.
>> Previous HEAD position was 6ee27c0... initial
>> Switched to branch 'master'
>> scons: Reading SConscript files ...
>> scons: done reading SConscript files.
>> scons: Building targets ...
>> g++ -o main.o -c main.cpp
>> scons: done building targets.
>> === .:
>> SConstruct: None
>> alpha.h: 3518d8c3310bba3c2a0da5489f569e14
>> beta.h: 2ff783593a2224d0574181661ab5f1b7
>> gamma.h: 2ff783593a2224d0574181661ab5f1b7
>> main: b53741b91f13474487bbbc1efe5e328b
>> main.cpp: ebf971b7da9c98f91067a9c601e96259
>> main.o: faaa740a3704b2c8f034bafd7c1722ff
>> === /usr/bin:
>> g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
>> 3518d8c3310bba3c2a0da5489f569e14  alpha.h
>> bca4755df4cd2eef7d51bfb007c36dd1  beta.h
>> 2ff783593a2224d0574181661ab5f1b7  gamma.h
>> ebf971b7da9c98f91067a9c601e96259  main.cpp
>>
>>  Regards,
>>
>> Piotr
>>
>>
>>
>>
>>
>> 2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>
>> Piotr,
>>
>>
>>
>> That should be fine.  Try to keep the dataset as simple and lightweight
>> as possible, since a test case needs to be created from it.  If you only
>> need a single c-file with a few header includes to reproduce, then that
>> should be plenty.
>>
>>
>>
>> If you feel comfortable request a patch addition, you can do so here:
>> https://bitbucket.org/scons/scons.
>>
>>
>>
>> V/R,
>>
>> William
>>
>>
>>
>> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik <piotr.bartosik at gmail.com>
>> wrote:
>>
>> William,
>>
>> thank you for your answer. I'm stating to prepare a test data set. I'm
>> going to send you a compressed directory versioned with git (with .git dir
>> inside) and an exact list of steps to reproduce the problem. I hope that
>> this format is convenient to you, if not please let me know.
>>
>> Best regards,
>> Piotr
>>
>>
>>
>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>
>> Piotr,
>>
>>
>>
>> Also, as a side question, do you have a set of test case values we can
>> use for this case?
>>
>>
>>
>> V/R,
>>
>> William
>>
>>
>>
>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins <wblevins001 at gmail.com>
>> wrote:
>>
>> Piotr,
>>
>>
>>
>> I am sure you already did your homework, and we appreciate your input.
>> Before we start looking at this closely, can you guarantee that the
>> following files were not accidentally committed your GIT repository: object
>> files, libraries, executables, the .sconsign.dblite file itself?  I have
>> seen cases in the past were files like these get committed accidentally,
>> and that might cause effects like those you described.
>>
>>
>>
>> V/R,
>>
>> William
>>
>>
>>
>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik <
>> piotr.bartosik at gmail.com> wrote:
>>
>>       Hi,
>>
>> while using SCons (version 2.3.0) together with git I encountered some
>> problems with MD5 calculations. After rebasing to different revisions of my
>> repository I noticed strange cache misses (the proper .o files should be
>> already in the cache while they were actually rebuilt). This led me to the
>> conclusion that some MD5 values may be calculated incorrectly. In fact,
>> after using the script sconsign.py it became apparent that some MD5s in
>> .sconsign.dblite were incorrect (the sums calculated directly using hashlib
>> were different).
>>
>> To sum up, after this simple procedure:
>>
>> rm .sconsign.dblite
>>
>> git checkout SOME_VERSION_1
>>
>> scons
>>
>> git checkout SOME_VERSION_2
>>
>> scons
>>
>> sconsign.py reported incorrect MD5 sums.
>>
>> As it turned out, the problem is localized in Node/__init__.py. The code
>>
>> then.extend([None] * diff)
>>
>> in line 1050 causes desynchronization between two lists: the current
>> dependencies and info nodes for dependencies from the previous build
>> (containing, among others, previously calculated MD5 sums).
>>
>>
>>
>> For example, consider that in SOME_VERSION_1 a file has five dependencies:
>>
>> A B C D E
>>
>> and in SOME_VERSION_2 a new dependency appears:
>>
>> A B C X D E
>>
>> After line 1050, the lists "then" and "children" look like below:
>>
>> then:     A B C D E None
>>
>> children: A B C X D E
>>
>> Code further below updates children information. As it appears, such
>> desynchronization does not cause any problems if we are checking MD5s each
>> time: they just will be recalculated for X, D and E. But if we also look at
>> the timestamps, the condition of timestamp remaining the same allows SCons
>> to copy the MD5 sum from the "then" array to the current child object.
>> Consequently, if D and E have the same modification date, E's checksum will
>> be incorrectly copied into D's info. And this is exactly the error that I
>> observe.
>>
>> My quick fix builds "then" list in a proper way, .i.e. without
>> desynchronization (please see the attachment):
>>
>> then:     A B C None D E
>>
>> children: A B C X    D E
>>
>> Please look at it and tell me what you think. I don't know the SCons'
>> sources very well so my fix may be a little naive but this is what I can
>> provide for a start.
>>
>> Thanks,
>>
>> Piotr
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141024/97cebffa/attachment-0001.html>