[Scons-users] MD5-timestamp decider overrides csigs fromSConsign incorrectly

[Plunket, Tom]

Whoa, the list extender doesn’t do the right thing at all if the number of items has shrunk, either. It tries to append a negative number of items to the ‘this’ list. (I wonder if this is part of my “fixing missing dependencies by removing them” problem that I’ve seen.)

From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Andrew C. Morrow
Sent: Tuesday, February 23, 2016 9:09 AM
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] MD5-timestamp decider overrides csigs from SConsign incorrectly


If that is in fact the underlying issue, and with the qualification that I really don't want to re-ignite an old controversy, I would be very happy to see that issue addressed.

Thanks,
Andrew

On Thu, Jan 14, 2016 at 8:14 PM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:
See if this looks like your topic: http://scons.tigris.org/issues/show_bug.cgi?id=2980

On Fri, Jan 15, 2016 at 1:09 AM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:
This has already been reported I believe; let me lookup the issue.

On Fri, Jan 15, 2016 at 12:44 AM, Sean Houghton <sean.houghton at gmail.com<mailto:sean.houghton at gmail.com>> wrote:
We’ve been having a difficult-to-track-down issue which we believe boils down to the behavior of the MD5-timestamp decider when the list of dependencies has changed between current runs and previous runs.

The main Node.changed function calls the decider function for every input to a node in all cases (called out by a comment “Note that we now *always* check every dependency”). The inputs are determined by zip()ing together two lists: a list of current children and a list of previous ninfos. This happens even if the length of the list or the order of dependencies in the list has changed:

    diff = len(children) - len(then)
    if diff:
        # The old and new dependency lists are different lengths.
        # This always indicates that the Node must be rebuilt.
        # We also extend the old dependency list with enough None
        # entries to equal the new dependency list, for the benefit
        # of the loop below that updates node information.
        then.extend([None] * diff)
        if t: Trace(': old %s new %s' % (len(then), len(children)))
        result = True

    for child, prev_ni in zip(children, then):
        if child.changed_since_last_build(self, prev_ni):
            if t: Trace(': %s changed' % child)
            result = True

The implementation of `changed_since_last_build` on FS.File nodes for MD5-timestamp overwrites the csig of a node whenever the timestamp matches:

    def changed_timestamp_then_content(self, target, prev_ni):
        if not self.changed_timestamp_match(target, prev_ni):
            try:
                self.get_ninfo().csig = prev_ni.csig
            except AttributeError:
                pass
            return False
        return self.changed_content(target, prev_ni)

If `prev_ni` does not correspond to the current node, the timestamps are checked anyway. If the timestamps happen to match (one second granularity is easily coarse enough for this to be possible), then the node’s current csig is blindly overwritten by the csig pulled from `prev_ni`--even if it belongs to a completely different node!

In our setup, which uses CacheDir, this led to bad csigs and a bad “cachedir_bsig”, which led to retrieving the wrong file from the cache directory when the set of dependencies for a node changed.

Can someone confirm this a SCons bug? We're switching back to the basic MD5 decider for now. We'll also spend some time and attempt to fix it as soon as possible. (Note that we’re on 2.3.6, but I checked the BitBucket repo and the current version appears to have the same issue.)


_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users



_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users



IMPORTANT CONFIDENTIALITY NOTICE:

This E-mail(including any documents referred to in, or attached, to this E-mail) may contain information that is personal, confidential or the subject of copyright or other proprietary rights in favor of Aristocrat, its affiliates or third parties. This E-mail is intended only for the named addressee. Any privacy, confidence, copyright or other proprietary rights in favor of Aristocrat, its affiliates or third parties, is not lost because this E-mail was sent to you by mistake.

If you received this E-mail by mistake you should: (i) not copy, disclose, distribute or otherwise use it, or its contents, without the consent of Aristocrat or the owner of the relevant rights; (ii) let us know of the mistake by reply E-mail or by telephone (US 1-877-274-9661, or  AU +61 2 9013 6000); and (iii) delete it from your system and destroy all copies.

Any personal information contained in this E-mail must be handled in accordance with applicable privacy laws.

Electronic and internet communications can be interfered with or affected by viruses and other defects. As a result, such communications may not be successfully received or, if received, may cause interference with the integrity of receiving, processing or related systems (including hardware, software and data or information on, or using, that hardware or software). Aristocrat gives no assurances in relation to these matters.

If you have any doubts about the veracity or integrity of any electronic communication we appear to have sent you, please call (US 1-877-274-9661, or  AU +61 2 9013 6000) for clarification.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160227/95a7e6d6/attachment-0001.html>