[Scons-users] Bug causing corruption of .sconsign.dblite

[Andrew C. Morrow]

How frustrating.

This appears to have been a very passive aggressive way of
communicating that the SCons developers will not even consider fixing
this bug themselves.

I really wish that I had time to write my own fix for every bug I
encounter in every open source project I use, but I don't, and I don't
think anyone else does either. Additionally, project maintainers are
almost always able to design and implement a correct fix with less
overall effort from all involved due to their familiarity with project
internals and workflows.

Asking the existing project developers to consider scheduling a
confirmed bug into an upcoming release is not unreasonable. As a long
term SCons user, it is disappointing to me that a friendly ping about
a real bug would generate this sort of reply.


On Mon, Aug 24, 2015 at 5:44 PM, Andrew C. Morrow
<andrew.c.morrow at gmail.com> wrote:
> Unfortunately, no. I can't commit to doing that work right now.
>
>
> On Mon, Aug 24, 2015 at 5:06 PM, William Blevins <wblevins001 at gmail.com> wrote:
>> Let me answer your question with another.  Are you offering to contribute a
>> fully working patch?
>>
>>
>> On Aug 24, 2015 12:48 PM, "Andrew C. Morrow" <andrew.c.morrow at gmail.com>
>> wrote:
>>>
>>> I just was bitten by this bug during a bisect. The MD5-Timestamp
>>> decider is very useful for iterative developer builds so I usually
>>> have it enabled. Similar for CacheDir, especially during a bisect
>>> since a particular source file version may be revisited several times.
>>>
>>> It is unfortunate that these features interact poorly.
>>>
>>> Is there any chance of getting this issue on the roadmap to be fixed
>>> in SCons 2.4?
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
>>> On Fri, Oct 24, 2014 at 6:47 PM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>> > Andrew,
>>> >
>>> > Once there is a fix, then a release is inevitable. There isn't an
>>> > expected
>>> > completion date.
>>> >
>>> > I cannot speak for everyone else, but I feel that this is a lower
>>> > priority
>>> > issue, so I will not investigate further until I have finished
>>> > cross-language scanner enhancements.  If you would like this resolved
>>> > ASAP,
>>> > please feel free to contribute a finalized patch with tests.
>>> >
>>> > V/R,
>>> > William
>>> >
>>> > On Fri, Oct 24, 2014 at 10:21 AM, Andrew C. Morrow
>>> > <andrew.c.morrow at gmail.com> wrote:
>>> >>
>>> >>
>>> >> Hi -
>>> >>
>>> >> Is there a possibility for a 2.3.5 release containing a fix for this,
>>> >> once
>>> >> one is completed?
>>> >>
>>> >> Thanks,
>>> >> Andrew
>>> >>
>>> >>
>>> >> On Mon, Oct 20, 2014 at 10:20 AM, William Blevins
>>> >> <wblevins001 at gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Stijn,
>>> >>>
>>> >>> Negative.  The issue has nothing to do with GIT.
>>> >>>
>>> >>> The bug exists within the "MD5-timestamp" decider.  This decider can
>>> >>> falsely mark Nodes as "not up-to-date" because of how the dependency
>>> >>> cache
>>> >>> is expanded when new dependencies are added to a Node.  In worst case,
>>> >>> sometimes code will get compiled that wasn't necessary, but only after
>>> >>> the
>>> >>> timestamp on those files would change; thus, it's a performance bug,
>>> >>> but in
>>> >>> most cases will still be faster than using the "MD5" decider since it
>>> >>> doesn't need to compute the MD5 sums every pass.
>>> >>>
>>> >>> V/R,
>>> >>> William
>>> >>>
>>> >>> On Mon, Oct 20, 2014 at 4:31 AM, Stijn De Ruyck
>>> >>> <Stijn.DeRuyck at onsemi.com> wrote:
>>> >>>>
>>> >>>> I’m trying to understand this issue. Can someone summarize?
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> Does it mean Scons 2.3.0 is essentially unusable when combined with
>>> >>>> git?
>>> >>>>
>>> >>>> I’m currently converting 2 software projects to Scons, but I’m stuck
>>> >>>> with 2.3.0 because of Python dependencies…
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> Any checkout, pull, rebase, reset, … can corrupt the .sconsign.dblite
>>> >>>> file?
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> Best regards,
>>> >>>>
>>> >>>> Stijn
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of
>>> >>>> William Blevins
>>> >>>> Sent: Sunday, October 19, 2014 9:21 PM
>>> >>>> To: SCons users mailing list
>>> >>>> Subject: Re: [Scons-users] Bug causing corruption of .sconsign.dblite
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> Piotr,
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> I applied the patch to the latest SCons source, and it broke several
>>> >>>> automated tests.  I do not know whether those tests simply need to be
>>> >>>> updated or additional modifications are required.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> I am currently working on another issue regarding cross-language
>>> >>>> scanner
>>> >>>> support, and I cannot dedicate the time to investigate at present.
>>> >>>> If you
>>> >>>> are eager to have this fixed in the next SCons update, consider
>>> >>>> forking a
>>> >>>> repository @ https://bitbucket.org/scons/scons and creating a pull
>>> >>>> request.
>>> >>>> Additional developer notes can be found @ http://www.scons.org/wiki
>>> >>>> and
>>> >>>> there are individuals who can help if you have additional questions.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> V/R,
>>> >>>>
>>> >>>> William
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Fri, Oct 17, 2014 at 8:37 PM, William Blevins
>>> >>>> <wblevins001 at gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>> Piotr,
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> A software failure indeed.  After your test steps, the md5sum for
>>> >>>> beta.h
>>> >>>> is incorrect.  If I update the timestamp for beta.h, main.cpp will
>>> >>>> incorrectly rebuild.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> Thank you for a streamlined test case. I made a issue tracker for
>>> >>>> this
>>> >>>> defect: http://scons.tigris.org/issues/show_bug.cgi?id=2980
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> V/R,
>>> >>>>
>>> >>>> William
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Thu, Oct 16, 2014 at 7:58 PM, William Blevins
>>> >>>> <wblevins001 at gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>> Piotr,
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> Thanks for supplying the additional information. I will make my best
>>> >>>> effort to take a close look at the information this weekend.  In the
>>> >>>> unlikely event that a SCons developer hasn't gotten back to you by
>>> >>>> end of
>>> >>>> next week, please ping us with a gentle reminder.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> V/R,
>>> >>>>
>>> >>>> William
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik
>>> >>>> <piotr.bartosik at gmail.com> wrote:
>>> >>>>
>>> >>>> William,
>>> >>>>
>>> >>>> please find attached a simple test that reproduces the bug. I tested
>>> >>>> it
>>> >>>> on Linux. Below is my example output with discrepancies highlighted:
>>> >>>>
>>> >>>> $ ./test.sh ~/tools/scons
>>> >>>> Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
>>> >>>>
>>> >>>> You are in 'detached HEAD' state. You can look around, make
>>> >>>> experimental
>>> >>>> changes and commit them, and you can discard any commits you make in
>>> >>>> this
>>> >>>> state without impacting any branches by performing another checkout.
>>> >>>>
>>> >>>> If you want to create a new branch to retain commits you create, you
>>> >>>> may
>>> >>>> do so (now or later) by using -b with the checkout command again.
>>> >>>> Example:
>>> >>>>
>>> >>>>   git checkout -b new_branch_name
>>> >>>>
>>> >>>> HEAD is now at 6ee27c0... initial
>>> >>>> scons: Reading SConscript files ...
>>> >>>> scons: done reading SConscript files.
>>> >>>> scons: Building targets ...
>>> >>>> g++ -o main.o -c main.cpp
>>> >>>> g++ -o main main.o
>>> >>>> scons: done building targets.
>>> >>>> Previous HEAD position was 6ee27c0... initial
>>> >>>> Switched to branch 'master'
>>> >>>> scons: Reading SConscript files ...
>>> >>>> scons: done reading SConscript files.
>>> >>>> scons: Building targets ...
>>> >>>> g++ -o main.o -c main.cpp
>>> >>>> scons: done building targets.
>>> >>>> === .:
>>> >>>> SConstruct: None
>>> >>>> alpha.h: 3518d8c3310bba3c2a0da5489f569e14
>>> >>>> beta.h: 2ff783593a2224d0574181661ab5f1b7
>>> >>>> gamma.h: 2ff783593a2224d0574181661ab5f1b7
>>> >>>> main: b53741b91f13474487bbbc1efe5e328b
>>> >>>> main.cpp: ebf971b7da9c98f91067a9c601e96259
>>> >>>> main.o: faaa740a3704b2c8f034bafd7c1722ff
>>> >>>> === /usr/bin:
>>> >>>> g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
>>> >>>> 3518d8c3310bba3c2a0da5489f569e14  alpha.h
>>> >>>> bca4755df4cd2eef7d51bfb007c36dd1  beta.h
>>> >>>> 2ff783593a2224d0574181661ab5f1b7  gamma.h
>>> >>>> ebf971b7da9c98f91067a9c601e96259  main.cpp
>>> >>>>
>>> >>>> Regards,
>>> >>>>
>>> >>>> Piotr
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> 2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>> >>>>
>>> >>>> Piotr,
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> That should be fine.  Try to keep the dataset as simple and
>>> >>>> lightweight
>>> >>>> as possible, since a test case needs to be created from it.  If you
>>> >>>> only
>>> >>>> need a single c-file with a few header includes to reproduce, then
>>> >>>> that
>>> >>>> should be plenty.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> If you feel comfortable request a patch addition, you can do so here:
>>> >>>> https://bitbucket.org/scons/scons.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> V/R,
>>> >>>>
>>> >>>> William
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik
>>> >>>> <piotr.bartosik at gmail.com> wrote:
>>> >>>>
>>> >>>> William,
>>> >>>>
>>> >>>> thank you for your answer. I'm stating to prepare a test data set.
>>> >>>> I'm
>>> >>>> going to send you a compressed directory versioned with git (with
>>> >>>> .git dir
>>> >>>> inside) and an exact list of steps to reproduce the problem. I hope
>>> >>>> that
>>> >>>> this format is convenient to you, if not please let me know.
>>> >>>>
>>> >>>> Best regards,
>>> >>>> Piotr
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
>>> >>>>
>>> >>>> Piotr,
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> Also, as a side question, do you have a set of test case values we
>>> >>>> can
>>> >>>> use for this case?
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> V/R,
>>> >>>>
>>> >>>> William
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins
>>> >>>> <wblevins001 at gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>> Piotr,
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> I am sure you already did your homework, and we appreciate your
>>> >>>> input.
>>> >>>> Before we start looking at this closely, can you guarantee that the
>>> >>>> following files were not accidentally committed your GIT repository:
>>> >>>> object
>>> >>>> files, libraries, executables, the .sconsign.dblite file itself?  I
>>> >>>> have
>>> >>>> seen cases in the past were files like these get committed
>>> >>>> accidentally, and
>>> >>>> that might cause effects like those you described.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> V/R,
>>> >>>>
>>> >>>> William
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik
>>> >>>> <piotr.bartosik at gmail.com> wrote:
>>> >>>>
>>> >>>> Hi,
>>> >>>>
>>> >>>> while using SCons (version 2.3.0) together with git I encountered
>>> >>>> some
>>> >>>> problems with MD5 calculations. After rebasing to different revisions
>>> >>>> of my
>>> >>>> repository I noticed strange cache misses (the proper .o files should
>>> >>>> be
>>> >>>> already in the cache while they were actually rebuilt). This led me
>>> >>>> to the
>>> >>>> conclusion that some MD5 values may be calculated incorrectly. In
>>> >>>> fact,
>>> >>>> after using the script sconsign.py it became apparent that some MD5s
>>> >>>> in
>>> >>>> .sconsign.dblite were incorrect (the sums calculated directly using
>>> >>>> hashlib
>>> >>>> were different).
>>> >>>>
>>> >>>> To sum up, after this simple procedure:
>>> >>>>
>>> >>>> rm .sconsign.dblite
>>> >>>>
>>> >>>> git checkout SOME_VERSION_1
>>> >>>>
>>> >>>> scons
>>> >>>>
>>> >>>> git checkout SOME_VERSION_2
>>> >>>>
>>> >>>> scons
>>> >>>>
>>> >>>> sconsign.py reported incorrect MD5 sums.
>>> >>>>
>>> >>>> As it turned out, the problem is localized in Node/__init__.py. The
>>> >>>> code
>>> >>>>
>>> >>>> then.extend([None] * diff)
>>> >>>>
>>> >>>> in line 1050 causes desynchronization between two lists: the current
>>> >>>> dependencies and info nodes for dependencies from the previous build
>>> >>>> (containing, among others, previously calculated MD5 sums).
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> For example, consider that in SOME_VERSION_1 a file has five
>>> >>>> dependencies:
>>> >>>>
>>> >>>> A B C D E
>>> >>>>
>>> >>>> and in SOME_VERSION_2 a new dependency appears:
>>> >>>>
>>> >>>> A B C X D E
>>> >>>>
>>> >>>> After line 1050, the lists "then" and "children" look like below:
>>> >>>>
>>> >>>> then:     A B C D E None
>>> >>>>
>>> >>>> children: A B C X D E
>>> >>>>
>>> >>>> Code further below updates children information. As it appears, such
>>> >>>> desynchronization does not cause any problems if we are checking MD5s
>>> >>>> each
>>> >>>> time: they just will be recalculated for X, D and E. But if we also
>>> >>>> look at
>>> >>>> the timestamps, the condition of timestamp remaining the same allows
>>> >>>> SCons
>>> >>>> to copy the MD5 sum from the "then" array to the current child
>>> >>>> object.
>>> >>>> Consequently, if D and E have the same modification date, E's
>>> >>>> checksum will
>>> >>>> be incorrectly copied into D's info. And this is exactly the error
>>> >>>> that I
>>> >>>> observe.
>>> >>>>
>>> >>>> My quick fix builds "then" list in a proper way, .i.e. without
>>> >>>> desynchronization (please see the attachment):
>>> >>>>
>>> >>>> then:     A B C None D E
>>> >>>>
>>> >>>> children: A B C X    D E
>>> >>>>
>>> >>>> Please look at it and tell me what you think. I don't know the SCons'
>>> >>>> sources very well so my fix may be a little naive but this is what I
>>> >>>> can
>>> >>>> provide for a start.
>>> >>>>
>>> >>>> Thanks,
>>> >>>>
>>> >>>> Piotr
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Scons-users mailing list
>>> >>>> Scons-users at scons.org
>>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Scons-users mailing list
>>> >>>> Scons-users at scons.org
>>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Scons-users mailing list
>>> >>>> Scons-users at scons.org
>>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Scons-users mailing list
>>> >>>> Scons-users at scons.org
>>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Scons-users mailing list
>>> >>>> Scons-users at scons.org
>>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Scons-users mailing list
>>> >>>> Scons-users at scons.org
>>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>>>
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> Scons-users mailing list
>>> >>> Scons-users at scons.org
>>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Scons-users mailing list
>>> >> Scons-users at scons.org
>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Scons-users mailing list
>>> > Scons-users at scons.org
>>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>