[Scons-users] Making SCons deal more robustly with incremental linking

[William Blevins]

Looks like the issue exists in class Node method changed().

The function does not evaluate whether its signature has changed. At a
glance, I'm not sure how to perform the check, but it should be simple
since the md5sum or other metadata should already be stored somewhere...

V/R,
William

On Tue, Jun 9, 2015 at 7:30 PM, William Blevins <wblevins001 at gmail.com>
wrote:

>
>
> On Mon, Jun 8, 2015 at 4:26 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
>
>>
>> On Mon, Jun 8, 2015 at 2:37 PM, Evan Driscoll <driscoll at cs.wisc.edu>
>> wrote:
>>
>>> OK, so this is a bit of a weird request maybe.
>>>
>>> I am trying to add an SCons flag that will cause builds with GCC to use
>>> incremental linking. This boils down to adding "-fuse-ld=gold
>>> -fno-use-linker-plugin -Wl,--incremental" to LINKFLAGS, but I don't think
>>> that's so relevant.
>>>
>>> When this flag is present, I would like to make two enhancements:
>>> 1. Every file on which the linker is invoked should be marked as Precious
>>> 2. Every file on which the linker is invoked should have itself marked
>>> as a dependency
>>>
>>> For my purposes, a "mostly-there" solution that only is called for
>>> programs and not shared libraries would be fine. I also am not averse to
>>> modifying the scons source; we have a local version with some reasonably
>>> heavy patches anyway.
>>>
>>>
>>> In case it's not clear, what I mean by #2 is the following: the output
>>> of the linker depends not only on the .o files listed on the link line, but
>>> on the initial state of the exe, so the has of the exe before the link
>>> should contribute to the signature of the build step. Without some explicit
>>> action that I'm not sure what to do, the following sequence will cause a
>>> mis-build:
>>>  1. Build under scons
>>>  2. Modify an object file outside of scons
>>>  3. Relink outside of scons
>>>  4. Revert the object file to what it was
>>>  5. Rebuild with scons -- it will not relink because it thinks nothing
>>> has changed
>>> This sequence of actions would presumably be rare, but is serious if it
>>> happens: the result could easily be extremely hard to diagnose, and,
>>> because we share an scons cache between different working copies and users,
>>> a bad build could poison multiple builds. (This is enough of a concern we
>>> would quite possibly disable caching for linked files.)
>>>
>>
>> In this case, the md5sum of the resulting exe was equivalent to its
>> previous (SCons compiled) value; otherwise, I believe SCons would rebuild
>> the executable.
>>
>> I don't believe that performing incremental linking will result in a
>> different executable md5sum versus traditional linking for shared libraries
>> due to usage of "-fpic" (IE. PIC compliance) by default in SCons shared
>> library building.
>>
>> If the md5sums differ, please explain which files clearly.
>>
>
> You are right.  In the case that an item is modified, the item itself will
> not be rebuilt.  If I make a sample SConstruct, generate a small
> executable, and hand modify the executable, SCons still shows up-to-date
> despite the md5sum in .sconsign.dblite being different from the actual.
>
> Perhaps nodes should implicitly depend on themselves.
>
>
>>
>>>
>>
>> I tried setting PROGEMITTER to something simple that adds the resulting
>>> exe to the input list, but SCons complained about a circular dependency.
>>> The main other thing that comes to mind is somewhat of a hack:
>>>  1. For each linked file foo, make something like
>>>       Command('foo.hash', 'foo', 'md5sum $SOURCE > $TARGET')
>>>  2. During SConscript running, read foo.hash, put its contents into a
>>> Value, and make foo depend on that Value
>>> but that seems a bit ugly... and I'd still have to know how to
>>> automatically do that for each linked file.
>>>
>>
>> As long as you a using an md5sum-based decider (the SCons default), then
>> Nodes already depend on md5sums and those sums exist in .sconsdb.sqlite
>> file (might not have remembered the exact filename) in the same directory
>> as your SConstruct.  You can use the sconsign scripts to get the md5sums
>> from the database file.
>>
>>
>>>
>>> I can also imagine that a scanner could come in handy here, but that's
>>> an area I'm basically unfamiliar with. And the man page says that it should
>>> return a list of "File Nodes", meaning that I don't know how to get it to
>>> return something not in the file system.
>>>
>>>
>>> (Finally, just as a note: I think I haven't completely solved the
>>> problem even with the above. Gold will embed a Build ID in the resulting
>>> exe that will change each time, which means that with a solution to #2
>>> SCons would always consider it out-of-date. You can specify how it
>>> generates the Build ID, but I haven't gotten it to work quite right yet.)
>>>
>>> Evan
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>
>> V/R,
>> William
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150610/9a6b982a/attachment.html>