[Scons-users] Making SCons deal more robustly with incremental linking
William Blevins
wblevins001 at gmail.com
Fri Jun 19 02:16:46 EDT 2015
Team,
I ran my patch against a couple different setups including Dirk's
testsuite, and I think the performance loss is negligible at best, but Dirk
may feel otherwise. I did find a bug in my patch, since I wasn't checking
for attributeerror on the first pass.
diff -r c482dddeafc5 src/engine/SCons/Node/__init__.py
> --- a/src/engine/SCons/Node/__init__.py Wed May 20 21:59:19 2015 +0200
> +++ b/src/engine/SCons/Node/__init__.py Fri Jun 19 01:37:07 2015 -0400
> @@ -1077,9 +1077,15 @@
> if node is None:
> node = self
>
> - result = False
> + bi = node.get_stored_info().binfo
> + ni = node.get_stored_info().ninfo
>
> - bi = node.get_stored_info().binfo
> + try:
> + result = node.get_csig() != ni.csig
> + if result and t: Trace('csig modified.')
> + except AttributeError:
> + result = False
> +
> then = bi.bsourcesigs + bi.bdependsigs + bi.bimplicitsigs
> children = self.children()
>
I think this has value since SCons is focused on correct builds. Any
thoughts?
V/R,
William
On Sun, Jun 14, 2015 at 11:23 AM, William Blevins <wblevins001 at gmail.com>
wrote:
> Evan,
>
> Did this patch solve your issue? I am still waiting to see if I should
> create a pull request. I am considering doing so regardless, but it would
> be nice to get a thumbs up from a real world build.
>
> The performance loss scales to log(N) * md5sum which doesn't affect the
> performance O(n), but it might detect some subtle issue that are hard find
> otherwise.
>
> V/R,
> William
>
> On Thu, Jun 11, 2015 at 11:36 AM, Kenny, Jason L <jason.l.kenny at intel.com>
> wrote:
>
>> I would agree,
>>
>>
>>
>> I only pointing this out and an issue that might come up next once the
>> “get a link that should link to link” is fixed. We then get the common link
>> that should not need to link. I am sure people have seen this, it more of
>> an issue when you have large builds with lots of linking in it.
>>
>>
>>
>> Jason
>>
>>
>>
>>
>>
>> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf Of
>> *William Blevins
>> *Sent:* Thursday, June 11, 2015 10:16 AM
>>
>> *To:* SCons users mailing list
>> *Subject:* Re: [Scons-users] Making SCons deal more robustly with
>> incremental linking
>>
>>
>>
>> Jason,
>>
>>
>>
>> I was only trying to cover issue #2. I assume incremental linking
>> support is a rather large effort.
>>
>>
>>
>> V/R,
>>
>> William
>>
>>
>>
>> On Thu, Jun 11, 2015 at 10:39 AM, Kenny, Jason L <jason.l.kenny at intel.com>
>> wrote:
>>
>> I should note form an incremental liking point of view the main failure
>> I have seen with this is that an object file we rebuilt, but it results in
>> the same exact file. SCons will relink/rebuild the binary because of some
>> logic that just assumes that If the child changes the parent has to as
>> well. But it a common case people add comments or something to have results
>> in the same output. This is where SCons could skip the rebuilt in many
>> cases as this results in the same output. This is not 100% true as some
>> builder output might have timestamp info, etc added to the output based on
>> when it ran as well. The current logic covers this case, it would be nice
>> if we could do something to get the first case right as well as when you
>> have 100 so files and the one change on the bottom can cause 100 relinks
>> that are not needed when the rebuilt .o results in the same output.
>>
>>
>>
>> This is a bit the reverse of the current issue.
>>
>>
>>
>> Jason
>>
>>
>>
>> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf Of
>> *William Blevins
>> *Sent:* Thursday, June 11, 2015 9:33 AM
>>
>>
>> *To:* SCons users mailing list
>> *Subject:* Re: [Scons-users] Making SCons deal more robustly with
>> incremental linking
>>
>>
>>
>> I also thought that SCons would rebuild. It didn't. I made a patch that
>> should fix the issue if we decide that it should be fixed.
>>
>>
>>
>> V/R,
>>
>> William
>>
>>
>>
>> On Thu, Jun 11, 2015 at 10:31 AM, Kenny, Jason L <jason.l.kenny at intel.com>
>> wrote:
>>
>> I think got that, I thought SCons would have seen this as inconstant
>> and rebuilt.
>>
>>
>>
>> Jason
>>
>>
>>
>> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf Of
>> *William Blevins
>> *Sent:* Thursday, June 11, 2015 9:27 AM
>> *To:* SCons users mailing list
>>
>>
>> *Subject:* Re: [Scons-users] Making SCons deal more robustly with
>> incremental linking
>>
>>
>>
>> Jason,
>>
>>
>>
>> In his example, he compiled outside the SCons builder, then reverted the
>> modified object and library files to their previous state leaving the leaf
>> node (E.G. an compiled executable objects -> libraries -> exe) with a
>> different md5sum; thus, the ninfo.csig for the node as listed in the scons
>> database is different than the csig of the node on the filesystem.
>>
>>
>>
>> The patch I supplied previously would allow SCons to check if these leaf
>> nodes are modified outside the SCons context.
>>
>>
>>
>> V/R,
>>
>> William
>>
>>
>>
>> On Thu, Jun 11, 2015 at 10:13 AM, Kenny, Jason L <jason.l.kenny at intel.com>
>> wrote:
>>
>> I am confused on this .
>>
>> A leaf node did not change. How was this done? Normally this is done by
>> creating a hashsum and comparing that it is the same with the one stored.
>> How is this not recreated and still correct? Just because the children did
>> not change does not mean the given file did not change.
>>
>> Jason
>>
>>
>> -----Original Message-----
>> From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of
>> Dirk Bächle
>> Sent: Thursday, June 11, 2015 8:32 AM
>> To: scons-users at scons.org
>> Subject: Re: [Scons-users] Making SCons deal more robustly with
>> incremental linking
>>
>> Hi,
>>
>> On 10.06.2015 07:52, William Blevins wrote:
>> > Looks like the issue exists in class Node method changed().
>> >
>> > The function does not evaluate whether its signature has changed. At a
>> > glance, I'm not sure how to perform the check, but it should be simple
>> since the md5sum or other metadata should already be stored somewhere...
>> >
>>
>> it's a rather simple check indeed, which is omitted for improved speed
>> performance. It's assumed that SCons has full control over the build and
>> that all dependencies are correctly tracked...so when no children changed
>> why rebuilding the hashsum for the target?
>> It *must* be the same as before if it exists. ;)
>>
>> The basic problems here are the cyclic dependency based on incremental
>> linking (this is exactly why Gold adds a build ID to each target, I guess)
>> and trying to allow "external linking" by manually calling command lines
>> (?). SCons can't account for what people are doing outside of its scope...I
>> can only imagine a pseudo-Builder to work, which would have to relink files
>> automatically until all symbols are resolved (similar to the LaTeX PDF
>> Builder).
>>
>> Best regards,
>>
>> Dirk
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150619/110178a7/attachment.html>
More information about the Scons-users
mailing list