[Scons-users] Bug causing corruption of .sconsign.dblite

[Andrew C. Morrow]

This issue came up again for me, and I have a question. Several questions
actually.

As originally framed, it sounded like the issue was specific to using the
MD5-Timestamp decider, but the bug itself seems more general. I've read
http://scons.tigris.org/issues/show_bug.cgi?id=2980 a few times, and I've
also looked at the changed_timestamp_then_content function, and it just
isn't clear to me at all what the connection is between the linked ticket
and that Decider in particular.

Is it true that the bug only affects the MD5-Timestamp decider? If so, why,
and why aren't the other built-in Deciders affected? If you are writing
your own Decider, what should you avoid doing to be also affected? Or, if
this issue is not limited to the MD5-Timestamp decider, what is the actual
scope of the issue? If the other built-in Deciders are affected, why does
this not cause more problems in practice?

Thanks,
Andrew


On Wed, Aug 26, 2015 at 12:47 AM, Bill Deegan <bill at baddogconsulting.com>
wrote:

> Andrew,
>
>
> On Tue, Aug 25, 2015 at 12:54 PM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>> Dirk, Bill, William -
>>
>> Thanks for providing a great deal more context. I certainly understand
>> the need to prioritize features over bugs that have workarounds, and
>> it appears that I misconstrued the intent of William's earlier email,
>> so, apologies for that.
>>
>> Part of my frustration was that I actually agree with your assessment
>> that fixing this issue is perhaps very challenging. It did not seem
>> like something that would be reasonably tackled by anyone outside of
>> the core group of SCons developers, and I'm not one of those.
>>
>> As to the impact of the bug, using the MD5-Timestamp provider and
>> CacheDir each has substantial cost savings for us in terms of
>> incremental build times, so we do rely on them for developer
>> productivity. Being unable to use them together is really unfortunate.
>> While I appreciate Bill's point that spurious rebuilds of files is
>> 'harmless' in a technical sense, a single spuriously recompiled
>> translation unit early in the graph can cause a substantial amount of
>> follow-on work, in particular, very expensive re-links.
>>
>
> Perhaps better to describe this bug as not harmless, but not producing an
> incorrect build.
> Understood that the downstream results of the bug can be non-trivial
> amount of wasted compute and CPU.
>
>
>>
>> Finally, I'd actually be personally happy to help the SCons project
>> more with development, but my work commitments don't realistically
>> allow it. As you note, MongoDB is heavily dependent on SCons, and we
>> have a number of changes that we would like to see. As I'm sure is
>> unsurprising though, we have our development priorities too. We
>> actually have had some internal discussion recently about whether
>> MongoDB might sponsor some SCons development work on features or fixes
>> that are important to us. Would there be any interest from the core
>> developers?
>>
>
> Indeed yes.
> I do have a consulting business and bugfixing for open source projects is
> a non-trivial part thereof.
> Feel free to contact me off list.  Note that I'm currently bouncing back
> and forth between NYC and SF.
> Indeed mongo's NYC office is 3 blocks from our NYC apartment.
>
> -Bill
>
>
>>
>> Thanks,
>> Andrew
>>
>>
>> On Tue, Aug 25, 2015 at 3:04 PM, Dirk Bächle <tshortik at gmx.de> wrote:
>> > Hi Andrew,
>> >
>> > I'd like to second Bill's thoughts. The signature subsystem in SCons
>> isn't
>> > easy to comprehend and fix. It's scattered all over the place and pretty
>> > much undocumented (at least compared to the rest of our documentation,
>> it's
>> > far behind).
>> > So chances are slim to none that some newbie will jump on this issue and
>> > come up with a valid fix (not breaking any existing behaviour) in a
>> week or
>> > so.
>> >
>> > In fact, I did have a look at the problem some time ago and can't
>> recall all
>> > the details...but this isn't a straight-forward patch to make. It
>> probably
>> > involves changing the Decider API at some places, breaking of existing
>> > scripts included. There may be a way around this, but so far I found
>> none.
>> > I also didn't find any time to properly document or publish my findings
>> up
>> > to now...not to speak of improving the signature system documentation
>> along
>> > the way (which is also on my TODO list).
>> >
>> > I plan to pick up my work on this, but before that I have several other
>> > topics to fight down...just like every other (core) developer.
>> >
>> > Finally, this bug is connected to the "MD5-timestamp" decider only...so
>> a
>> > valid workaround exists in using the "MD5" decider, which should make
>> things
>> > work fine again. I understand that this gives you headaches for the
>> MongoDB
>> > project, regarding build performance. But we currently have some larger
>> > topics (slots, subprocess wrapper, new toolchains, Python 3) to address,
>> > that seem strategically important for the project. I hope you can
>> understand
>> > that we, as core team, see it as our duty to attack these things first.
>> >
>> > Best regards,
>> >
>> > Dirk
>> >
>> > On 25.08.2015 16:33, Bill Deegan wrote:
>> >>
>> >> Andrew,
>> >>
>> >> Generally SCons releases are a compilation of bug fixes submitted by
>> our
>> >> user community.
>> >> Nobody works on SCons full time.
>> >>
>> >> So scheduling a bug for a given release implies that there's a
>> guarantee
>> >> that there will be a fix developed and/or means that a
>> >> given release cannot be released without that fix.
>> >>
>> >> As far as I know, no one is actively working on this particular bug, so
>> >> holding up a release for this bug to be fixed would mean
>> >> that other fixes wouldn't make there way to our user community.
>> >>
>> >> So while I may have phrased what William Blevins said differently,
>> there's
>> >> some truth in his message.
>> >> If you want a bug fixed you have three options: 1) fix it your self, 2)
>> >> encourage another community member to fix it for you, 3) pay
>> >> someone to fix it for you.
>> >> This is true with all open source (with the possible exception of
>> projects
>> >> where there is a commercial entity behind it).
>> >>
>> >> Also it's fair to add that this particular bug may potential require a
>> >> non-trivial amount of code change and understanding of SCons
>> >> internals. Then we'll have to make sure we have test coverage for the
>> bug
>> >> as well.
>> >>
>> >> If you'll note currently we're working on getting the slots changes
>> >> stabilized tested, and hopefully sometime soon, released.  This
>> >> change provides significant runtime and memory improvements.
>> >>
>> >> If it a choice between getting that out and fixing a bug where the
>> failure
>> >> is that some objects are rebuilt when they don't need to
>> >> be, it seems a pretty clear choice as to which we should focus on.
>> >>
>> >> Now if the bug were inverted, and objects were not getting built when
>> they
>> >> should, then this would be a much higher priority bug and
>> >> likely would get more immediate attention.
>> >>
>> >> I don't think anyone has said they will not work on this bug, but they
>> >> have said they have higher priority issues they are working
>> >> on. Of course the bug which is bugging you now will always seem the
>> >> highest priority..
>> >>
>> >> Hope that helps!
>> >> -Bill
>> >>
>> >
>> > _______________________________________________
>> > Scons-users mailing list
>> > Scons-users at scons.org
>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160512/b2256c02/attachment.html>