[Scons-users] Bug causing corruption of .sconsign.dblite

[Andrew C. Morrow]

Unfortunately, no. I can't commit to doing that work right now.


On Mon, Aug 24, 2015 at 5:06 PM, William Blevins <wblevins001 at gmail.com> wrote:
> Let me answer your question with another.  Are you offering to contribute a
> fully working patch?
>
>
> On Aug 24, 2015 12:48 PM, "Andrew C. Morrow" <andrew.c.morrow at gmail.com>
> wrote:
>>
>> I just was bitten by this bug during a bisect. The MD5-Timestamp
>> decider is very useful for iterative developer builds so I usually
>> have it enabled. Similar for CacheDir, especially during a bisect
>> since a particular source file version may be revisited several times.
>>
>> It is unfortunate that these features interact poorly.
>>
>> Is there any chance of getting this issue on the roadmap to be fixed
>> in SCons 2.4?
>>
>> Thanks,
>> Andrew
>>
>>
>> On Fri, Oct 24, 2014 at 6:47 PM, William Blevins <wblevins001 at gmail.com>
>> wrote:
>> > Andrew,
>> >
>> > Once there is a fix, then a release is inevitable. There isn't an
>> > expected
>> > completion date.
>> >
>> > I cannot speak for everyone else, but I feel that this is a lower
>> > priority
>> > issue, so I will not investigate further until I have finished
>> > cross-language scanner enhancements.  If you would like this resolved
>> > ASAP,
>> > please feel free to contribute a finalized patch with tests.
>> >
>> > V/R,
>> > William
>> >
>> > On Fri, Oct 24, 2014 at 10:21 AM, Andrew C. Morrow
>> > <andrew.c.morrow at gmail.com> wrote:
>> >>
>> >>
>> >> Hi -
>> >>
>> >> Is there a possibility for a 2.3.5 release containing a fix for this,
>> >> once
>> >> one is completed?
>> >>
>> >> Thanks,
>> >> Andrew
>> >>
>> >>
>> >> On Mon, Oct 20, 2014 at 10:20 AM, William Blevins
>> >> <wblevins001 at gmail.com>
>> >> wrote:
>> >>>
>> >>> Stijn,
>> >>>
>> >>> Negative.  The issue has nothing to do with GIT.
>> >>>
>> >>> The bug exists within the "MD5-timestamp" decider.  This decider can
>> >>> falsely mark Nodes as "not up-to-date" because of how the dependency
>> >>> cache
>> >>> is expanded when new dependencies are added to a Node.  In worst case,
>> >>> sometimes code will get compiled that wasn't necessary, but only after
>> >>> the
>> >>> timestamp on those files would change; thus, it's a performance bug,
>> >>> but in
>> >>> most cases will still be faster than using the "MD5" decider since it
>> >>> doesn't need to compute the MD5 sums every pass.
>> >>>
>> >>> V/R,
>> >>> William
>> >>>
>> >>> On Mon, Oct 20, 2014 at 4:31 AM, Stijn De Ruyck
>> >>> <Stijn.DeRuyck at onsemi.com> wrote:
>> >>>>
>> >>>> I’m trying to understand this issue. Can someone summarize?
>> >>>>
>> >>>>
>> >>>>
>> >>>> Does it mean Scons 2.3.0 is essentially unusable when combined with
>> >>>> git?
>> >>>>
>> >>>> I’m currently converting 2 software projects to Scons, but I’m stuck
>> >>>> with 2.3.0 because of Python dependencies…
>> >>>>
>> >>>>
>> >>>>
>> >>>> Any checkout, pull, rebase, reset, … can corrupt the .sconsign.dblite
>> >>>> file?
>> >>>>
>> >>>>
>> >>>>
>> >>>> Best regards,
>> >>>>
>> >>>> Stijn
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of
>> >>>> William Blevins
>> >>>> Sent: Sunday, October 19, 2014 9:21 PM
>> >>>> To: SCons users mailing list
>> >>>> Subject: Re: [Scons-users] Bug causing corruption of .sconsign.dblite
>> >>>>
>> >>>>
>> >>>>
>> >>>> Piotr,
>> >>>>
>> >>>>
>> >>>>
>> >>>> I applied the patch to the latest SCons source, and it broke several
>> >>>> automated tests.  I do not know whether those tests simply need to be
>> >>>> updated or additional modifications are required.
>> >>>>
>> >>>>
>> >>>>
>> >>>> I am currently working on another issue regarding cross-language
>> >>>> scanner
>> >>>> support, and I cannot dedicate the time to investigate at present.
>> >>>> If you
>> >>>> are eager to have this fixed in the next SCons update, consider
>> >>>> forking a
>> >>>> repository @ https://bitbucket.org/scons/scons and creating a pull
>> >>>> request.
>> >>>> Additional developer notes can be found @ http://www.scons.org/wiki
>> >>>> and
>> >>>> there are individuals who can help if you have additional questions.
>> >>>>
>> >>>>
>> >>>>
>> >>>> V/R,
>> >>>>
>> >>>> William
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Fri, Oct 17, 2014 at 8:37 PM, William Blevins
>> >>>> <wblevins001 at gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> Piotr,
>> >>>>
>> >>>>
>> >>>>
>> >>>> A software failure indeed.  After your test steps, the md5sum for
>> >>>> beta.h
>> >>>> is incorrect.  If I update the timestamp for beta.h, main.cpp will
>> >>>> incorrectly rebuild.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Thank you for a streamlined test case. I made a issue tracker for
>> >>>> this
>> >>>> defect: http://scons.tigris.org/issues/show_bug.cgi?id=2980
>> >>>>
>> >>>>
>> >>>>
>> >>>> V/R,
>> >>>>
>> >>>> William
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Oct 16, 2014 at 7:58 PM, William Blevins
>> >>>> <wblevins001 at gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> Piotr,
>> >>>>
>> >>>>
>> >>>>
>> >>>> Thanks for supplying the additional information. I will make my best
>> >>>> effort to take a close look at the information this weekend.  In the
>> >>>> unlikely event that a SCons developer hasn't gotten back to you by
>> >>>> end of
>> >>>> next week, please ping us with a gentle reminder.
>> >>>>
>> >>>>
>> >>>>
>> >>>> V/R,
>> >>>>
>> >>>> William
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik
>> >>>> <piotr.bartosik at gmail.com> wrote:
>> >>>>
>> >>>> William,
>> >>>>
>> >>>> please find attached a simple test that reproduces the bug. I tested
>> >>>> it
>> >>>> on Linux. Below is my example output with discrepancies highlighted:
>> >>>>
>> >>>> $ ./test.sh ~/tools/scons
>> >>>> Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
>> >>>>
>> >>>> You are in 'detached HEAD' state. You can look around, make
>> >>>> experimental
>> >>>> changes and commit them, and you can discard any commits you make in
>> >>>> this
>> >>>> state without impacting any branches by performing another checkout.
>> >>>>
>> >>>> If you want to create a new branch to retain commits you create, you
>> >>>> may
>> >>>> do so (now or later) by using -b with the checkout command again.
>> >>>> Example:
>> >>>>
>> >>>>   git checkout -b new_branch_name
>> >>>>
>> >>>> HEAD is now at 6ee27c0... initial
>> >>>> scons: Reading SConscript files ...
>> >>>> scons: done reading SConscript files.
>> >>>> scons: Building targets ...
>> >>>> g++ -o main.o -c main.cpp
>> >>>> g++ -o main main.o
>> >>>> scons: done building targets.
>> >>>> Previous HEAD position was 6ee27c0... initial
>> >>>> Switched to branch 'master'
>> >>>> scons: Reading SConscript files ...
>> >>>> scons: done reading SConscript files.
>> >>>> scons: Building targets ...
>> >>>> g++ -o main.o -c main.cpp
>> >>>> scons: done building targets.
>> >>>> === .:
>> >>>> SConstruct: None
>> >>>> alpha.h: 3518d8c3310bba3c2a0da5489f569e14
>> >>>> beta.h: 2ff783593a2224d0574181661ab5f1b7
>> >>>> gamma.h: 2ff783593a2224d0574181661ab5f1b7
>> >>>> main: b53741b91f13474487bbbc1efe5e328b
>> >>>> main.cpp: ebf971b7da9c98f91067a9c601e96259
>> >>>> main.o: faaa740a3704b2c8f034bafd7c1722ff
>> >>>> === /usr/bin:
>> >>>> g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
>> >>>> 3518d8c3310bba3c2a0da5489f569e14  alpha.h
>> >>>> bca4755df4cd2eef7d51bfb007c36dd1  beta.h
>> >>>> 2ff783593a2224d0574181661ab5f1b7  gamma.h
>> >>>> ebf971b7da9c98f91067a9c601e96259  main.cpp
>> >>>>
>> >>>> Regards,
>> >>>>
>> >>>> Piotr
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> 2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>> >>>>
>> >>>> Piotr,
>> >>>>
>> >>>>
>> >>>>
>> >>>> That should be fine.  Try to keep the dataset as simple and
>> >>>> lightweight
>> >>>> as possible, since a test case needs to be created from it.  If you
>> >>>> only
>> >>>> need a single c-file with a few header includes to reproduce, then
>> >>>> that
>> >>>> should be plenty.
>> >>>>
>> >>>>
>> >>>>
>> >>>> If you feel comfortable request a patch addition, you can do so here:
>> >>>> https://bitbucket.org/scons/scons.
>> >>>>
>> >>>>
>> >>>>
>> >>>> V/R,
>> >>>>
>> >>>> William
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik
>> >>>> <piotr.bartosik at gmail.com> wrote:
>> >>>>
>> >>>> William,
>> >>>>
>> >>>> thank you for your answer. I'm stating to prepare a test data set.
>> >>>> I'm
>> >>>> going to send you a compressed directory versioned with git (with
>> >>>> .git dir
>> >>>> inside) and an exact list of steps to reproduce the problem. I hope
>> >>>> that
>> >>>> this format is convenient to you, if not please let me know.
>> >>>>
>> >>>> Best regards,
>> >>>> Piotr
>> >>>>
>> >>>>
>> >>>>
>> >>>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>> >>>>
>> >>>> Piotr,
>> >>>>
>> >>>>
>> >>>>
>> >>>> Also, as a side question, do you have a set of test case values we
>> >>>> can
>> >>>> use for this case?
>> >>>>
>> >>>>
>> >>>>
>> >>>> V/R,
>> >>>>
>> >>>> William
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins
>> >>>> <wblevins001 at gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> Piotr,
>> >>>>
>> >>>>
>> >>>>
>> >>>> I am sure you already did your homework, and we appreciate your
>> >>>> input.
>> >>>> Before we start looking at this closely, can you guarantee that the
>> >>>> following files were not accidentally committed your GIT repository:
>> >>>> object
>> >>>> files, libraries, executables, the .sconsign.dblite file itself?  I
>> >>>> have
>> >>>> seen cases in the past were files like these get committed
>> >>>> accidentally, and
>> >>>> that might cause effects like those you described.
>> >>>>
>> >>>>
>> >>>>
>> >>>> V/R,
>> >>>>
>> >>>> William
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik
>> >>>> <piotr.bartosik at gmail.com> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> while using SCons (version 2.3.0) together with git I encountered
>> >>>> some
>> >>>> problems with MD5 calculations. After rebasing to different revisions
>> >>>> of my
>> >>>> repository I noticed strange cache misses (the proper .o files should
>> >>>> be
>> >>>> already in the cache while they were actually rebuilt). This led me
>> >>>> to the
>> >>>> conclusion that some MD5 values may be calculated incorrectly. In
>> >>>> fact,
>> >>>> after using the script sconsign.py it became apparent that some MD5s
>> >>>> in
>> >>>> .sconsign.dblite were incorrect (the sums calculated directly using
>> >>>> hashlib
>> >>>> were different).
>> >>>>
>> >>>> To sum up, after this simple procedure:
>> >>>>
>> >>>> rm .sconsign.dblite
>> >>>>
>> >>>> git checkout SOME_VERSION_1
>> >>>>
>> >>>> scons
>> >>>>
>> >>>> git checkout SOME_VERSION_2
>> >>>>
>> >>>> scons
>> >>>>
>> >>>> sconsign.py reported incorrect MD5 sums.
>> >>>>
>> >>>> As it turned out, the problem is localized in Node/__init__.py. The
>> >>>> code
>> >>>>
>> >>>> then.extend([None] * diff)
>> >>>>
>> >>>> in line 1050 causes desynchronization between two lists: the current
>> >>>> dependencies and info nodes for dependencies from the previous build
>> >>>> (containing, among others, previously calculated MD5 sums).
>> >>>>
>> >>>>
>> >>>>
>> >>>> For example, consider that in SOME_VERSION_1 a file has five
>> >>>> dependencies:
>> >>>>
>> >>>> A B C D E
>> >>>>
>> >>>> and in SOME_VERSION_2 a new dependency appears:
>> >>>>
>> >>>> A B C X D E
>> >>>>
>> >>>> After line 1050, the lists "then" and "children" look like below:
>> >>>>
>> >>>> then:     A B C D E None
>> >>>>
>> >>>> children: A B C X D E
>> >>>>
>> >>>> Code further below updates children information. As it appears, such
>> >>>> desynchronization does not cause any problems if we are checking MD5s
>> >>>> each
>> >>>> time: they just will be recalculated for X, D and E. But if we also
>> >>>> look at
>> >>>> the timestamps, the condition of timestamp remaining the same allows
>> >>>> SCons
>> >>>> to copy the MD5 sum from the "then" array to the current child
>> >>>> object.
>> >>>> Consequently, if D and E have the same modification date, E's
>> >>>> checksum will
>> >>>> be incorrectly copied into D's info. And this is exactly the error
>> >>>> that I
>> >>>> observe.
>> >>>>
>> >>>> My quick fix builds "then" list in a proper way, .i.e. without
>> >>>> desynchronization (please see the attachment):
>> >>>>
>> >>>> then:     A B C None D E
>> >>>>
>> >>>> children: A B C X    D E
>> >>>>
>> >>>> Please look at it and tell me what you think. I don't know the SCons'
>> >>>> sources very well so my fix may be a little naive but this is what I
>> >>>> can
>> >>>> provide for a start.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Piotr
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Scons-users mailing list
>> >>>> Scons-users at scons.org
>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Scons-users mailing list
>> >>>> Scons-users at scons.org
>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Scons-users mailing list
>> >>>> Scons-users at scons.org
>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Scons-users mailing list
>> >>>> Scons-users at scons.org
>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Scons-users mailing list
>> >>>> Scons-users at scons.org
>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Scons-users mailing list
>> >>>> Scons-users at scons.org
>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Scons-users mailing list
>> >>> Scons-users at scons.org
>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> Scons-users mailing list
>> >> Scons-users at scons.org
>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>
>> >
>> >
>> > _______________________________________________
>> > Scons-users mailing list
>> > Scons-users at scons.org
>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>