[Scons-users] Making SCons deal more robustly with incremental linking

[William Blevins]

On Mon, Jun 8, 2015 at 4:26 PM, William Blevins <wblevins001 at gmail.com>
wrote:

>
> On Mon, Jun 8, 2015 at 2:37 PM, Evan Driscoll <driscoll at cs.wisc.edu>
> wrote:
>
>> OK, so this is a bit of a weird request maybe.
>>
>> I am trying to add an SCons flag that will cause builds with GCC to use
>> incremental linking. This boils down to adding "-fuse-ld=gold
>> -fno-use-linker-plugin -Wl,--incremental" to LINKFLAGS, but I don't think
>> that's so relevant.
>>
>> When this flag is present, I would like to make two enhancements:
>> 1. Every file on which the linker is invoked should be marked as Precious
>> 2. Every file on which the linker is invoked should have itself marked as
>> a dependency
>>
>> For my purposes, a "mostly-there" solution that only is called for
>> programs and not shared libraries would be fine. I also am not averse to
>> modifying the scons source; we have a local version with some reasonably
>> heavy patches anyway.
>>
>>
>> In case it's not clear, what I mean by #2 is the following: the output of
>> the linker depends not only on the .o files listed on the link line, but on
>> the initial state of the exe, so the has of the exe before the link should
>> contribute to the signature of the build step. Without some explicit action
>> that I'm not sure what to do, the following sequence will cause a mis-build:
>>  1. Build under scons
>>  2. Modify an object file outside of scons
>>  3. Relink outside of scons
>>  4. Revert the object file to what it was
>>  5. Rebuild with scons -- it will not relink because it thinks nothing
>> has changed
>> This sequence of actions would presumably be rare, but is serious if it
>> happens: the result could easily be extremely hard to diagnose, and,
>> because we share an scons cache between different working copies and users,
>> a bad build could poison multiple builds. (This is enough of a concern we
>> would quite possibly disable caching for linked files.)
>>
>
> In this case, the md5sum of the resulting exe was equivalent to its
> previous (SCons compiled) value; otherwise, I believe SCons would rebuild
> the executable.
>
> I don't believe that performing incremental linking will result in a
> different executable md5sum versus traditional linking for shared libraries
> due to usage of "-fpic" (IE. PIC compliance) by default in SCons shared
> library building.
>
> If the md5sums differ, please explain which files clearly.
>

You are right.  In the case that an item is modified, the item itself will
not be rebuilt.  If I make a sample SConstruct, generate a small
executable, and hand modify the executable, SCons still shows up-to-date
despite the md5sum in .sconsign.dblite being different from the actual.

Perhaps nodes should implicitly depend on themselves.


>
>>
>
> I tried setting PROGEMITTER to something simple that adds the resulting
>> exe to the input list, but SCons complained about a circular dependency.
>> The main other thing that comes to mind is somewhat of a hack:
>>  1. For each linked file foo, make something like
>>       Command('foo.hash', 'foo', 'md5sum $SOURCE > $TARGET')
>>  2. During SConscript running, read foo.hash, put its contents into a
>> Value, and make foo depend on that Value
>> but that seems a bit ugly... and I'd still have to know how to
>> automatically do that for each linked file.
>>
>
> As long as you a using an md5sum-based decider (the SCons default), then
> Nodes already depend on md5sums and those sums exist in .sconsdb.sqlite
> file (might not have remembered the exact filename) in the same directory
> as your SConstruct.  You can use the sconsign scripts to get the md5sums
> from the database file.
>
>
>>
>> I can also imagine that a scanner could come in handy here, but that's an
>> area I'm basically unfamiliar with. And the man page says that it should
>> return a list of "File Nodes", meaning that I don't know how to get it to
>> return something not in the file system.
>>
>>
>> (Finally, just as a note: I think I haven't completely solved the problem
>> even with the above. Gold will embed a Build ID in the resulting exe that
>> will change each time, which means that with a solution to #2 SCons would
>> always consider it out-of-date. You can specify how it generates the Build
>> ID, but I haven't gotten it to work quite right yet.)
>>
>> Evan
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
> V/R,
> William
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150609/fd46c0f3/attachment.html>