[Scons-users] Bug causing corruption of .sconsign.dblite

Mon Aug 24 17:06:31 EDT 2015

Let me answer your question with another.  Are you offering to contribute a
fully working patch?
On Aug 24, 2015 12:48 PM, "Andrew C. Morrow" <andrew.c.morrow at gmail.com>
wrote:
>
> I just was bitten by this bug during a bisect. The MD5-Timestamp
> decider is very useful for iterative developer builds so I usually
> have it enabled. Similar for CacheDir, especially during a bisect
> since a particular source file version may be revisited several times.
>
> It is unfortunate that these features interact poorly.
>
> Is there any chance of getting this issue on the roadmap to be fixed
> in SCons 2.4?
>
> Thanks,
> Andrew
>
>
> On Fri, Oct 24, 2014 at 6:47 PM, William Blevins <wblevins001 at gmail.com>
wrote:
> > Andrew,
> >
> > Once there is a fix, then a release is inevitable. There isn't an
expected
> > completion date.
> >
> > I cannot speak for everyone else, but I feel that this is a lower
priority
> > issue, so I will not investigate further until I have finished
> > cross-language scanner enhancements.  If you would like this resolved
ASAP,
> > please feel free to contribute a finalized patch with tests.
> >
> > V/R,
> > William
> >
> > On Fri, Oct 24, 2014 at 10:21 AM, Andrew C. Morrow
> > <andrew.c.morrow at gmail.com> wrote:
> >>
> >>
> >> Hi -
> >>
> >> Is there a possibility for a 2.3.5 release containing a fix for this,
once
> >> one is completed?
> >>
> >> Thanks,
> >> Andrew
> >>
> >>
> >> On Mon, Oct 20, 2014 at 10:20 AM, William Blevins <
wblevins001 at gmail.com>
> >> wrote:
> >>>
> >>> Stijn,
> >>>
> >>> Negative.  The issue has nothing to do with GIT.
> >>>
> >>> The bug exists within the "MD5-timestamp" decider.  This decider can
> >>> falsely mark Nodes as "not up-to-date" because of how the dependency
cache
> >>> is expanded when new dependencies are added to a Node.  In worst case,
> >>> sometimes code will get compiled that wasn't necessary, but only
after the
> >>> timestamp on those files would change; thus, it's a performance bug,
but in
> >>> most cases will still be faster than using the "MD5" decider since it
> >>> doesn't need to compute the MD5 sums every pass.
> >>>
> >>> V/R,
> >>> William
> >>>
> >>> On Mon, Oct 20, 2014 at 4:31 AM, Stijn De Ruyck
> >>> <Stijn.DeRuyck at onsemi.com> wrote:
> >>>>
> >>>> I’m trying to understand this issue. Can someone summarize?
> >>>>
> >>>>
> >>>>
> >>>> Does it mean Scons 2.3.0 is essentially unusable when combined with
git?
> >>>>
> >>>> I’m currently converting 2 software projects to Scons, but I’m stuck
> >>>> with 2.3.0 because of Python dependencies…
> >>>>
> >>>>
> >>>>
> >>>> Any checkout, pull, rebase, reset, … can corrupt the .sconsign.dblite
> >>>> file?
> >>>>
> >>>>
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Stijn
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of
> >>>> William Blevins
> >>>> Sent: Sunday, October 19, 2014 9:21 PM
> >>>> To: SCons users mailing list
> >>>> Subject: Re: [Scons-users] Bug causing corruption of .sconsign.dblite
> >>>>
> >>>>
> >>>>
> >>>> Piotr,
> >>>>
> >>>>
> >>>>
> >>>> I applied the patch to the latest SCons source, and it broke several
> >>>> automated tests.  I do not know whether those tests simply need to be
> >>>> updated or additional modifications are required.
> >>>>
> >>>>
> >>>>
> >>>> I am currently working on another issue regarding cross-language
scanner
> >>>> support, and I cannot dedicate the time to investigate at present.
If you
> >>>> are eager to have this fixed in the next SCons update, consider
forking a
> >>>> repository @ https://bitbucket.org/scons/scons and creating a pull
request.
> >>>> Additional developer notes can be found @ http://www.scons.org/wiki
and
> >>>> there are individuals who can help if you have additional questions.
> >>>>
> >>>>
> >>>>
> >>>> V/R,
> >>>>
> >>>> William
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Oct 17, 2014 at 8:37 PM, William Blevins <
wblevins001 at gmail.com>
> >>>> wrote:
> >>>>
> >>>> Piotr,
> >>>>
> >>>>
> >>>>
> >>>> A software failure indeed.  After your test steps, the md5sum for
beta.h
> >>>> is incorrect.  If I update the timestamp for beta.h, main.cpp will
> >>>> incorrectly rebuild.
> >>>>
> >>>>
> >>>>
> >>>> Thank you for a streamlined test case. I made a issue tracker for
this
> >>>> defect: http://scons.tigris.org/issues/show_bug.cgi?id=2980
> >>>>
> >>>>
> >>>>
> >>>> V/R,
> >>>>
> >>>> William
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Oct 16, 2014 at 7:58 PM, William Blevins <
wblevins001 at gmail.com>
> >>>> wrote:
> >>>>
> >>>> Piotr,
> >>>>
> >>>>
> >>>>
> >>>> Thanks for supplying the additional information. I will make my best
> >>>> effort to take a close look at the information this weekend.  In the
> >>>> unlikely event that a SCons developer hasn't gotten back to you by
end of
> >>>> next week, please ping us with a gentle reminder.
> >>>>
> >>>>
> >>>>
> >>>> V/R,
> >>>>
> >>>> William
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Oct 16, 2014 at 10:40 AM, Piotr Bartosik
> >>>> <piotr.bartosik at gmail.com> wrote:
> >>>>
> >>>> William,
> >>>>
> >>>> please find attached a simple test that reproduces the bug. I tested
it
> >>>> on Linux. Below is my example output with discrepancies highlighted:
> >>>>
> >>>> $ ./test.sh ~/tools/scons
> >>>> Note: checking out '6ee27c0db68515e3faa71bf25c0ed7a883ecf697'.
> >>>>
> >>>> You are in 'detached HEAD' state. You can look around, make
experimental
> >>>> changes and commit them, and you can discard any commits you make in
> >>>> this
> >>>> state without impacting any branches by performing another checkout.
> >>>>
> >>>> If you want to create a new branch to retain commits you create, you
may
> >>>> do so (now or later) by using -b with the checkout command again.
> >>>> Example:
> >>>>
> >>>>   git checkout -b new_branch_name
> >>>>
> >>>> HEAD is now at 6ee27c0... initial
> >>>> scons: Reading SConscript files ...
> >>>> scons: done reading SConscript files.
> >>>> scons: Building targets ...
> >>>> g++ -o main.o -c main.cpp
> >>>> g++ -o main main.o
> >>>> scons: done building targets.
> >>>> Previous HEAD position was 6ee27c0... initial
> >>>> Switched to branch 'master'
> >>>> scons: Reading SConscript files ...
> >>>> scons: done reading SConscript files.
> >>>> scons: Building targets ...
> >>>> g++ -o main.o -c main.cpp
> >>>> scons: done building targets.
> >>>> === .:
> >>>> SConstruct: None
> >>>> alpha.h: 3518d8c3310bba3c2a0da5489f569e14
> >>>> beta.h: 2ff783593a2224d0574181661ab5f1b7
> >>>> gamma.h: 2ff783593a2224d0574181661ab5f1b7
> >>>> main: b53741b91f13474487bbbc1efe5e328b
> >>>> main.cpp: ebf971b7da9c98f91067a9c601e96259
> >>>> main.o: faaa740a3704b2c8f034bafd7c1722ff
> >>>> === /usr/bin:
> >>>> g++: bb36ec5f9b8a8bfc02d1c4bf917a58e1
> >>>> 3518d8c3310bba3c2a0da5489f569e14  alpha.h
> >>>> bca4755df4cd2eef7d51bfb007c36dd1  beta.h
> >>>> 2ff783593a2224d0574181661ab5f1b7  gamma.h
> >>>> ebf971b7da9c98f91067a9c601e96259  main.cpp
> >>>>
> >>>> Regards,
> >>>>
> >>>> Piotr
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> 2014-10-15 16:00 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
> >>>>
> >>>> Piotr,
> >>>>
> >>>>
> >>>>
> >>>> That should be fine.  Try to keep the dataset as simple and
lightweight
> >>>> as possible, since a test case needs to be created from it.  If you
only
> >>>> need a single c-file with a few header includes to reproduce, then
that
> >>>> should be plenty.
> >>>>
> >>>>
> >>>>
> >>>> If you feel comfortable request a patch addition, you can do so here:
> >>>> https://bitbucket.org/scons/scons.
> >>>>
> >>>>
> >>>>
> >>>> V/R,
> >>>>
> >>>> William
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Oct 15, 2014 at 5:13 AM, Piotr Bartosik
> >>>> <piotr.bartosik at gmail.com> wrote:
> >>>>
> >>>> William,
> >>>>
> >>>> thank you for your answer. I'm stating to prepare a test data set.
I'm
> >>>> going to send you a compressed directory versioned with git (with
.git dir
> >>>> inside) and an exact list of steps to reproduce the problem. I hope
that
> >>>> this format is convenient to you, if not please let me know.
> >>>>
> >>>> Best regards,
> >>>> Piotr
> >>>>
> >>>>
> >>>>
> >>>> 2014-10-15 1:25 GMT+02:00 William Blevins <wblevins001 at gmail.com>:
> >>>>
> >>>> Piotr,
> >>>>
> >>>>
> >>>>
> >>>> Also, as a side question, do you have a set of test case values we
can
> >>>> use for this case?
> >>>>
> >>>>
> >>>>
> >>>> V/R,
> >>>>
> >>>> William
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Oct 14, 2014 at 7:23 PM, William Blevins <
wblevins001 at gmail.com>
> >>>> wrote:
> >>>>
> >>>> Piotr,
> >>>>
> >>>>
> >>>>
> >>>> I am sure you already did your homework, and we appreciate your
input.
> >>>> Before we start looking at this closely, can you guarantee that the
> >>>> following files were not accidentally committed your GIT repository:
object
> >>>> files, libraries, executables, the .sconsign.dblite file itself?  I
have
> >>>> seen cases in the past were files like these get committed
accidentally, and
> >>>> that might cause effects like those you described.
> >>>>
> >>>>
> >>>>
> >>>> V/R,
> >>>>
> >>>> William
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Oct 14, 2014 at 10:14 AM, Piotr Bartosik
> >>>> <piotr.bartosik at gmail.com> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> while using SCons (version 2.3.0) together with git I encountered
some
> >>>> problems with MD5 calculations. After rebasing to different
revisions of my
> >>>> repository I noticed strange cache misses (the proper .o files
should be
> >>>> already in the cache while they were actually rebuilt). This led me
to the
> >>>> conclusion that some MD5 values may be calculated incorrectly. In
fact,
> >>>> after using the script sconsign.py it became apparent that some MD5s
in
> >>>> .sconsign.dblite were incorrect (the sums calculated directly using
hashlib
> >>>> were different).
> >>>>
> >>>> To sum up, after this simple procedure:
> >>>>
> >>>> rm .sconsign.dblite
> >>>>
> >>>> git checkout SOME_VERSION_1
> >>>>
> >>>> scons
> >>>>
> >>>> git checkout SOME_VERSION_2
> >>>>
> >>>> scons
> >>>>
> >>>> sconsign.py reported incorrect MD5 sums.
> >>>>
> >>>> As it turned out, the problem is localized in Node/__init__.py. The
code
> >>>>
> >>>> then.extend([None] * diff)
> >>>>
> >>>> in line 1050 causes desynchronization between two lists: the current
> >>>> dependencies and info nodes for dependencies from the previous build
> >>>> (containing, among others, previously calculated MD5 sums).
> >>>>
> >>>>
> >>>>
> >>>> For example, consider that in SOME_VERSION_1 a file has five
> >>>> dependencies:
> >>>>
> >>>> A B C D E
> >>>>
> >>>> and in SOME_VERSION_2 a new dependency appears:
> >>>>
> >>>> A B C X D E
> >>>>
> >>>> After line 1050, the lists "then" and "children" look like below:
> >>>>
> >>>> then:     A B C D E None
> >>>>
> >>>> children: A B C X D E
> >>>>
> >>>> Code further below updates children information. As it appears, such
> >>>> desynchronization does not cause any problems if we are checking
MD5s each
> >>>> time: they just will be recalculated for X, D and E. But if we also
look at
> >>>> the timestamps, the condition of timestamp remaining the same allows
SCons
> >>>> to copy the MD5 sum from the "then" array to the current child
object.
> >>>> Consequently, if D and E have the same modification date, E's
checksum will
> >>>> be incorrectly copied into D's info. And this is exactly the error
that I
> >>>> observe.
> >>>>
> >>>> My quick fix builds "then" list in a proper way, .i.e. without
> >>>> desynchronization (please see the attachment):
> >>>>
> >>>> then:     A B C None D E
> >>>>
> >>>> children: A B C X    D E
> >>>>
> >>>> Please look at it and tell me what you think. I don't know the SCons'
> >>>> sources very well so my fix may be a little naive but this is what I
can
> >>>> provide for a start.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Piotr
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Scons-users mailing list
> >>>> Scons-users at scons.org
> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Scons-users mailing list
> >>>> Scons-users at scons.org
> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Scons-users mailing list
> >>>> Scons-users at scons.org
> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Scons-users mailing list
> >>>> Scons-users at scons.org
> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Scons-users mailing list
> >>>> Scons-users at scons.org
> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Scons-users mailing list
> >>>> Scons-users at scons.org
> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Scons-users mailing list
> >>> Scons-users at scons.org
> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>
> >>
> >>
> >> _______________________________________________
> >> Scons-users mailing list
> >> Scons-users at scons.org
> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>
> >
> >
> > _______________________________________________
> > Scons-users mailing list
> > Scons-users at scons.org
> > https://pairlist4.pair.net/mailman/listinfo/scons-users
> >
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150824/a5963ca0/attachment-0001.html>