[Scons-users] Bug causing corruption of .sconsign.dblite

Tue Aug 25 10:33:49 EDT 2015

Andrew,

Generally SCons releases are a compilation of bug fixes submitted by our
user community.
Nobody works on SCons full time.

So scheduling a bug for a given release implies that there's a guarantee
that there will be a fix developed and/or means that a given release cannot
be released without that fix.

As far as I know, no one is actively working on this particular bug, so
holding up a release for this bug to be fixed would mean that other fixes
wouldn't make there way to our user community.

So while I may have phrased what William Blevins said differently, there's
some truth in his message.
If you want a bug fixed you have three options: 1) fix it your self, 2)
encourage another community member to fix it for you, 3) pay someone to fix
it for you.
This is true with all open source (with the possible exception of projects
where there is a commercial entity behind it).

Also it's fair to add that this particular bug may potential require a
non-trivial amount of code change and understanding of SCons internals.
Then we'll have to make sure we have test coverage for the bug as well.

If you'll note currently we're working on getting the slots changes
stabilized tested, and hopefully sometime soon, released.  This change
provides significant runtime and memory improvements.

If it a choice between getting that out and fixing a bug where the failure
is that some objects are rebuilt when they don't need to be, it seems a
pretty clear choice as to which we should focus on.

Now if the bug were inverted, and objects were not getting built when they
should, then this would be a much higher priority bug and likely would get
more immediate attention.

I don't think anyone has said they will not work on this bug, but they have
said they have higher priority issues they are working on. Of course the
bug which is bugging you now will always seem the highest priority..

Hope that helps!
-Bill

On Tue, Aug 25, 2015 at 4:03 AM, Andrew C. Morrow <andrew.c.morrow at gmail.com
> wrote:

> How frustrating.
>
> This appears to have been a very passive aggressive way of
> communicating that the SCons developers will not even consider fixing
> this bug themselves.
>
> I really wish that I had time to write my own fix for every bug I
> encounter in every open source project I use, but I don't, and I don't
> think anyone else does either. Additionally, project maintainers are
> almost always able to design and implement a correct fix with less
> overall effort from all involved due to their familiarity with project
> internals and workflows.
>
> Asking the existing project developers to consider scheduling a
> confirmed bug into an upcoming release is not unreasonable. As a long
> term SCons user, it is disappointing to me that a friendly ping about
> a real bug would generate this sort of reply.
>
>
> On Mon, Aug 24, 2015 at 5:44 PM, Andrew C. Morrow
> <andrew.c.morrow at gmail.com> wrote:
> > Unfortunately, no. I can't commit to doing that work right now.
> >
> >
> > On Mon, Aug 24, 2015 at 5:06 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
> >> Let me answer your question with another.  Are you offering to
> contribute a
> >> fully working patch?
> >>
> >>
> >> On Aug 24, 2015 12:48 PM, "Andrew C. Morrow" <andrew.c.morrow at gmail.com
> >
> >> wrote:
> >>>
> >>> I just was bitten by this bug during a bisect. The MD5-Timestamp
> >>> decider is very useful for iterative developer builds so I usually
> >>> have it enabled. Similar for CacheDir, especially during a bisect
> >>> since a particular source file version may be revisited several times.
> >>>
> >>> It is unfortunate that these features interact poorly.
> >>>
> >>> Is there any chance of getting this issue on the roadmap to be fixed
> >>> in SCons 2.4?
> >>>
> >>> Thanks,
> >>> Andrew
> >>>
> >>>
> >>> On Fri, Oct 24, 2014 at 6:47 PM, William Blevins <
> wblevins001 at gmail.com>
> >>> wrote:
> >>> > Andrew,
> >>> >
> >>> > Once there is a fix, then a release is inevitable. There isn't an
> >>> > expected
> >>> > completion date.
> >>> >
> >>> > I cannot speak for everyone else, but I feel that this is a lower
> >>> > priority
> >>> > issue, so I will not investigate further until I have finished
> >>> > cross-language scanner enhancements.  If you would like this resolved
> >>> > ASAP,
> >>> > please feel free to contribute a finalized patch with tests.
> >>> >
> >>> > V/R,
> >>> > William
> >>> >
> >>> > On Fri, Oct 24, 2014 at 10:21 AM, Andrew C. Morrow
> >>> > <andrew.c.morrow at gmail.com> wrote:
> >>> >>
> >>> >>
> >>> >> Hi -
> >>> >>
> >>> >> Is there a possibility for a 2.3.5 release containing a fix for
> this,
> >>> >> once
> >>> >> one is completed?
> >>> >>
> >>> >> Thanks,
> >>> >> Andrew
> >>> >>
> >>> >>
> >>> >> On Mon, Oct 20, 2014 at 10:20 AM, William Blevins
> >>> >> <wblevins001 at gmail.com>
> >>> >> wrote:
> >>> >>>
> >>> >>> Stijn,
> >>> >>>
> >>> >>> Negative.  The issue has nothing to do with GIT.
> >>> >>>
> >>> >>> The bug exists within the "MD5-timestamp" decider.  This decider
> can
> >>> >>> falsely mark Nodes as "not up-to-date" because of how the
> dependency
> >>> >>> cache
> >>> >>> is expanded when new dependencies are added to a Node.  In worst
> case,
> >>> >>> sometimes code will get compiled that wasn't necessary, but only
> after
> >>> >>> the
> >>> >>> timestamp on those files would change; thus, it's a performance
> bug,
> >>> >>> but in
> >>> >>> most cases will still be faster than using the "MD5" decider since
> it
> >>> >>> doesn't need to compute the MD5 sums every pass.
> >>> >>>
> >>> >>> V/R,
> >>> >>> William
> >>> >>>
> >>> >>> On Mon, Oct 20, 2014 at 4:31 AM, Stijn De Ruyck
> >>> >>> <Stijn.DeRuyck at onsemi.com> wrote:
> >>> >>>>
> >>> >>>> I’m trying to understand this issue. Can someone summarize?
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> Does it mean Scons 2.3.0 is essentially unusable when combined
> with
> >>> >>>> git?
> >>> >>>>
> >>> >>>> I’m currently converting 2 software projects to Scons, but I’m
> stuck
> >>> >>>> with 2.3.0 because of Python dependencies…
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> Any checkout, pull, rebase, reset, … can corrupt the
> .sconsign.dblite
> >>> >>>> file?
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> Best regards,
> >>> >>>>
> >>> >>>> Stijn
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> From: Scons-users [mailto:scons-users-bounces at scons.org] On
> Behalf Of
> >>> >>>> William Blevins
> >>> >>>> Sent: Sunday, October 19, 2014 9:21 PM
> >>> >>>> To: SCons users mailing list
> >>> >>>> Subject: Re: [Scons-users] Bug causing corruption of
> .sconsign.dblite
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> Piotr,
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> I applied the patch to the latest SCons source, and it broke
> several
> >>> >>>> automated tests.  I do not know whether those tests simply need
> to be
> >>> >>>> updated or additional modifications are required.
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> I am currently working on another issue regarding cross-language
> >>> >>>> scanner
> >>> >>>> support, and I cannot dedicate the time to investigate at present.
> >>> >>>> If you
> >>> >>>> are eager to have this fixed in the next SCons update, consider
> >>> >>>> forking a
> >>> >>>> repository @ https://bitbucket.org/scons/scons and creating a
> pull
> >>> >>>> request.
> >>> >>>> Additional developer notes can be found @
> http://www.scons.org/wiki
> >>> >>>> and
> >>> >>>> there are individuals who can help if you have additional
> questions.
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> V/R,
> >>> >>>>
> >>> >>>> William
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> On Fri, Oct 17, 2014 at 8:37 PM, William Blevins
> >>> >>>> <wblevins001 at gmail.com>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>> Piotr,
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> A software failure indeed.  After your test steps, the md5sum for
> >>> >>>> beta.h
> >>> >>>> is incorrect.  If I update the timestamp for beta.h, main.cpp will
> >>> >>>> incorrectly rebuild.
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> Thank you for a streamlined test case. I made a issue tracker for
> >>> >>>> this
> >>> >>>> defect: http://scons.tigris.org/issues/show_bug.cgi?id=2980
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> V/R,
> >>> >>>>
> >>> >>>> William
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> On Thu, Oct 16, 2014 at 7:58 PM, William Blevins
> >>> >>>> <wblevins001 at gmail.com>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>> Piotr,
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> Thanks for supplying the additional information. I will make my
> best
> >>> >>>> effort to take a close look at the information this weekend.  In
> the
> >>> >>>> unlikely event that a SCons developer hasn't gotten back to you by
> >>> >>>> end of
> >>> >>>> next week, please ping us with a gentle reminder.
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> V/R,
> >>> >>>>
> >>> >>>> William
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik
> >>> >>>> <piotr.bartosik at gmail.com> wrote:
> >>> >>>>
> >>> >>>> William,
> >>> >>>>
> >>> >>>> please find attached a simple test that reproduces the bug. I
> tested
> >>> >>>> it
> >>> >>>> on Linux. Below is my example output with discrepancies
> highlighted:
> >>> >>>>
> >>> >>>> $ ./test.sh ~/tools/scons
> >>> >>>> Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
> >>> >>>>
> >>> >>>> You are in 'detached HEAD' state. You can look around, make
> >>> >>>> experimental
> >>> >>>> changes and commit them, and you can discard any commits you make
> in
> >>> >>>> this
> >>> >>>> state without impacting any branches by performing another
> checkout.
> >>> >>>>
> >>> >>>> If you want to create a new branch to retain commits you create,
> you
> >>> >>>> may
> >>> >>>> do so (now or later) by using -b with the checkout command again.
> >>> >>>> Example:
> >>> >>>>
> >>> >>>>   git checkout -b new_branch_name
> >>> >>>>
> >>> >>>> HEAD is now at 6ee27c0... initial
> >>> >>>> scons: Reading SConscript files ...
> >>> >>>> scons: done reading SConscript files.
> >>> >>>> scons: Building targets ...
> >>> >>>> g++ -o main.o -c main.cpp
> >>> >>>> g++ -o main main.o
> >>> >>>> scons: done building targets.
> >>> >>>> Previous HEAD position was 6ee27c0... initial
> >>> >>>> Switched to branch 'master'
> >>> >>>> scons: Reading SConscript files ...
> >>> >>>> scons: done reading SConscript files.
> >>> >>>> scons: Building targets ...
> >>> >>>> g++ -o main.o -c main.cpp
> >>> >>>> scons: done building targets.
> >>> >>>> === .:
> >>> >>>> SConstruct: None
> >>> >>>> alpha.h: 3518d8c3310bba3c2a0da5489f569e14
> >>> >>>> beta.h: 2ff783593a2224d0574181661ab5f1b7
> >>> >>>> gamma.h: 2ff783593a2224d0574181661ab5f1b7
> >>> >>>> main: b53741b91f13474487bbbc1efe5e328b
> >>> >>>> main.cpp: ebf971b7da9c98f91067a9c601e96259
> >>> >>>> main.o: faaa740a3704b2c8f034bafd7c1722ff
> >>> >>>> === /usr/bin:
> >>> >>>> g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
> >>> >>>> 3518d8c3310bba3c2a0da5489f569e14  alpha.h
> >>> >>>> bca4755df4cd2eef7d51bfb007c36dd1  beta.h
> >>> >>>> 2ff783593a2224d0574181661ab5f1b7  gamma.h
> >>> >>>> ebf971b7da9c98f91067a9c601e96259  main.cpp
> >>> >>>>
> >>> >>>> Regards,
> >>> >>>>
> >>> >>>> Piotr
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> 2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com
> >:
> >>> >>>>
> >>> >>>> Piotr,
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> That should be fine.  Try to keep the dataset as simple and
> >>> >>>> lightweight
> >>> >>>> as possible, since a test case needs to be created from it.  If
> you
> >>> >>>> only
> >>> >>>> need a single c-file with a few header includes to reproduce, then
> >>> >>>> that
> >>> >>>> should be plenty.
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> If you feel comfortable request a patch addition, you can do so
> here:
> >>> >>>> https://bitbucket.org/scons/scons.
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> V/R,
> >>> >>>>
> >>> >>>> William
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik
> >>> >>>> <piotr.bartosik at gmail.com> wrote:
> >>> >>>>
> >>> >>>> William,
> >>> >>>>
> >>> >>>> thank you for your answer. I'm stating to prepare a test data set.
> >>> >>>> I'm
> >>> >>>> going to send you a compressed directory versioned with git (with
> >>> >>>> .git dir
> >>> >>>> inside) and an exact list of steps to reproduce the problem. I
> hope
> >>> >>>> that
> >>> >>>> this format is convenient to you, if not please let me know.
> >>> >>>>
> >>> >>>> Best regards,
> >>> >>>> Piotr
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com
> >:
> >>> >>>>
> >>> >>>> Piotr,
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> Also, as a side question, do you have a set of test case values we
> >>> >>>> can
> >>> >>>> use for this case?
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> V/R,
> >>> >>>>
> >>> >>>> William
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins
> >>> >>>> <wblevins001 at gmail.com>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>> Piotr,
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> I am sure you already did your homework, and we appreciate your
> >>> >>>> input.
> >>> >>>> Before we start looking at this closely, can you guarantee that
> the
> >>> >>>> following files were not accidentally committed your GIT
> repository:
> >>> >>>> object
> >>> >>>> files, libraries, executables, the .sconsign.dblite file itself?
> I
> >>> >>>> have
> >>> >>>> seen cases in the past were files like these get committed
> >>> >>>> accidentally, and
> >>> >>>> that might cause effects like those you described.
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> V/R,
> >>> >>>>
> >>> >>>> William
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik
> >>> >>>> <piotr.bartosik at gmail.com> wrote:
> >>> >>>>
> >>> >>>> Hi,
> >>> >>>>
> >>> >>>> while using SCons (version 2.3.0) together with git I encountered
> >>> >>>> some
> >>> >>>> problems with MD5 calculations. After rebasing to different
> revisions
> >>> >>>> of my
> >>> >>>> repository I noticed strange cache misses (the proper .o files
> should
> >>> >>>> be
> >>> >>>> already in the cache while they were actually rebuilt). This led
> me
> >>> >>>> to the
> >>> >>>> conclusion that some MD5 values may be calculated incorrectly. In
> >>> >>>> fact,
> >>> >>>> after using the script sconsign.py it became apparent that some
> MD5s
> >>> >>>> in
> >>> >>>> .sconsign.dblite were incorrect (the sums calculated directly
> using
> >>> >>>> hashlib
> >>> >>>> were different).
> >>> >>>>
> >>> >>>> To sum up, after this simple procedure:
> >>> >>>>
> >>> >>>> rm .sconsign.dblite
> >>> >>>>
> >>> >>>> git checkout SOME_VERSION_1
> >>> >>>>
> >>> >>>> scons
> >>> >>>>
> >>> >>>> git checkout SOME_VERSION_2
> >>> >>>>
> >>> >>>> scons
> >>> >>>>
> >>> >>>> sconsign.py reported incorrect MD5 sums.
> >>> >>>>
> >>> >>>> As it turned out, the problem is localized in Node/__init__.py.
> The
> >>> >>>> code
> >>> >>>>
> >>> >>>> then.extend([None] * diff)
> >>> >>>>
> >>> >>>> in line 1050 causes desynchronization between two lists: the
> current
> >>> >>>> dependencies and info nodes for dependencies from the previous
> build
> >>> >>>> (containing, among others, previously calculated MD5 sums).
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> For example, consider that in SOME_VERSION_1 a file has five
> >>> >>>> dependencies:
> >>> >>>>
> >>> >>>> A B C D E
> >>> >>>>
> >>> >>>> and in SOME_VERSION_2 a new dependency appears:
> >>> >>>>
> >>> >>>> A B C X D E
> >>> >>>>
> >>> >>>> After line 1050, the lists "then" and "children" look like below:
> >>> >>>>
> >>> >>>> then:     A B C D E None
> >>> >>>>
> >>> >>>> children: A B C X D E
> >>> >>>>
> >>> >>>> Code further below updates children information. As it appears,
> such
> >>> >>>> desynchronization does not cause any problems if we are checking
> MD5s
> >>> >>>> each
> >>> >>>> time: they just will be recalculated for X, D and E. But if we
> also
> >>> >>>> look at
> >>> >>>> the timestamps, the condition of timestamp remaining the same
> allows
> >>> >>>> SCons
> >>> >>>> to copy the MD5 sum from the "then" array to the current child
> >>> >>>> object.
> >>> >>>> Consequently, if D and E have the same modification date, E's
> >>> >>>> checksum will
> >>> >>>> be incorrectly copied into D's info. And this is exactly the error
> >>> >>>> that I
> >>> >>>> observe.
> >>> >>>>
> >>> >>>> My quick fix builds "then" list in a proper way, .i.e. without
> >>> >>>> desynchronization (please see the attachment):
> >>> >>>>
> >>> >>>> then:     A B C None D E
> >>> >>>>
> >>> >>>> children: A B C X    D E
> >>> >>>>
> >>> >>>> Please look at it and tell me what you think. I don't know the
> SCons'
> >>> >>>> sources very well so my fix may be a little naive but this is
> what I
> >>> >>>> can
> >>> >>>> provide for a start.
> >>> >>>>
> >>> >>>> Thanks,
> >>> >>>>
> >>> >>>> Piotr
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> _______________________________________________
> >>> >>>> Scons-users mailing list
> >>> >>>> Scons-users at scons.org
> >>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> _______________________________________________
> >>> >>>> Scons-users mailing list
> >>> >>>> Scons-users at scons.org
> >>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> _______________________________________________
> >>> >>>> Scons-users mailing list
> >>> >>>> Scons-users at scons.org
> >>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> _______________________________________________
> >>> >>>> Scons-users mailing list
> >>> >>>> Scons-users at scons.org
> >>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> _______________________________________________
> >>> >>>> Scons-users mailing list
> >>> >>>> Scons-users at scons.org
> >>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> _______________________________________________
> >>> >>>> Scons-users mailing list
> >>> >>>> Scons-users at scons.org
> >>> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>>>
> >>> >>>
> >>> >>>
> >>> >>> _______________________________________________
> >>> >>> Scons-users mailing list
> >>> >>> Scons-users at scons.org
> >>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>>
> >>> >>
> >>> >>
> >>> >> _______________________________________________
> >>> >> Scons-users mailing list
> >>> >> Scons-users at scons.org
> >>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >>
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Scons-users mailing list
> >>> > Scons-users at scons.org
> >>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>> >
> >>> _______________________________________________
> >>> Scons-users mailing list
> >>> Scons-users at scons.org
> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>
> >>
> >> _______________________________________________
> >> Scons-users mailing list
> >> Scons-users at scons.org
> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150825/4b3e09fc/attachment-0001.html>