[Scons-users] Detecting changed targets

[William Blevins]

To clarify on broken tests. Some were actually broken, but most were just
written to bad assumptions. Like having builders that didn't actually write
any lead nodes to disk then checking if scons was up-to-date.

Yours seems a bit more intrusive, but that might be fine (or better). There
was a bit more code in the actual PR (versus the comment) but it was
comparable.

V/R,
William

On Wed, Jan 11, 2017 at 10:52 PM, William Blevins <wblevins001 at gmail.com>
wrote:

> Mark,
>
> I am on board with a change like this. It has been proposed or discussed
> several times. I dabbled with it a while back to see if it was feasible:
> https://bitbucket.org/scons/scons/pull-requests/240/added-
> support-for-detecting-corruption/diff
>
> Problem was that this broke tons of tests and I didn't get much in the way
> of enthusiasm at the time. How our the tests under your patch? You can run
> them like "./runtests.py -a -j <cpus>" from the repo.
>
> V/R,
> William
>
> On Wed, Jan 11, 2017 at 12:10 PM, Mark Terrel <mark at unbounded.systems>
> wrote:
>
>> Hi SCons folks,
>>
>> Now that the holidays are over, I was hoping someone might give me
>> feedback on my proposal.  Any thoughts?
>>
>> Thanks,
>> Mark
>>
>> On Wed, Dec 28, 2016 at 3:01 PM, Mark Terrel <mark at unbounded.systems>
>> wrote:
>>
>>> Hi SCons gurus!
>>>
>>> I've been a user of SCons since 2010 (at a previous company) and am
>>> using it for a new project at a new company now.  We're doing all builds
>>> and development in containers, so I've created a SCons builder to build
>>> Docker images.  (I'm happy to share that once it's had more testing.)
>>>  That's working great so far, except for one issue that I'd like to solve.
>>>
>>> The issue stems from the fact that Docker images are not stored in the
>>> filesystem; they're stored in a common object store repository and are
>>> tagged with a name.  So while a typical user of SCons has a reasonable
>>> expectation that no one is going to come modify their build output
>>> directory contents, it now becomes much more likely that Developer A builds
>>> a Docker image with tag "production_container" and Developer B could build
>>> a different image and overwrite (actually move the tag)
>>> "production_container" to point to a different Docker image.  This makes it
>>> harder to ensure correctness of builds because SCons does not currently
>>> rebuild a target if the signature of the target changes.
>>>
>>> I found this exchange
>>> <http://scons-users.scons.narkive.com/7BJnwEox/corrupted-objects-are-not-rebuilt-as-one-might-expect>
>>> on the scons-users list from ages ago, where Steven Knight suggested a new
>>> ValidateTargets option that would detect if a target was changed (or had
>>> been corrupted) unbeknownst to SCons and would then rebuild that target.
>>> So I took a shot at implementing that feature and would love some feedback
>>> on whether this is a good approach and whether, after it meets the dev
>>> team's approval, you might be willing to incorporate it.
>>>
>>> This commit
>>> <https://bitbucket.org/mterrel/scons/commits/1b0c0f22d13596f709b6e3ca64a8f2ef41fd4c11>contains
>>> the proposed feature as a proof of concept.  It has two new user-facing
>>> functions:
>>>
>>> *ValidateTargets('none' | 'content')*
>>>
>>> Sets the global default behavior for all target nodes.
>>> 'none' is the default and is the same as SCons' current behavior, which
>>> is to not check the signatures for nodes that are only targets.
>>>
>>> 'content' uses the existing content signatures to check whether a target
>>> node has been modified outside of SCons' knowledge.
>>>
>>> *env.ValidateTarget(target_list, 'none' | 'content')*
>>>
>>> Sets the validation behavior for specific node(s), overriding the global
>>> default behavior for those nodes.  Options 'none' and 'content' behave the
>>> same as above.
>>>
>>>
>>> ​I think this new behavior contributes toward one of SCons' goals of
>>> correctness of builds.  I think it would also be particularly useful for
>>> other kinds of targets that might be in shared storage, such as AWS S3 or
>>> other object storage.  Those types of output storage are becoming more and
>>> more common as developers start distributing and testing builds across
>>> cloud environments.
>>>
>>> ​So please take a look at the proposal and give me any feedback or ideas
>>> and let me know if this is something I should try to move forward on.
>>>
>>> Thanks!
>>> Mark
>>>
>>> P.S.  I tested this on top of 2.5.1 and am running regression tests now
>>> on tip.
>>>
>>> --
>>> Mark Terrel  |  *Unbounded Systems*
>>>
>>
>>
>>
>> --
>> Mark Terrel  |  *Unbounded Systems*
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170111/20f4a1ba/attachment.html>