[Scons-users] Performance of version 2.5.0 vs 2.3.0 on Windows host dropped significantly

Tue Aug 9 10:31:05 EDT 2016

Henrik,

As a side note, the performance differences from previous versions is
useful. Do you also happen to have performance for full builds for my
reference? Some of those patches may have traded base overhead (slower
already up-to-date) for faster runtime (full or partial build time).

V/R,
William

On Tue, Aug 9, 2016 at 3:21 PM, William Blevins <wblevins001 at gmail.com>
wrote:

> Henrik,
>
> Unfortunately, the 0.2s difference for 2.3.6 cannot be helped. The 2.5.0
> change didn't affect much code, but greatly affected implicit dependency
> scanning for the better. I don't know if it will affect you personally, but
> 2.5.0 added cross-language implicit scanning support.
>
> V/R,
> William
>
> On Tue, Aug 9, 2016 at 1:45 PM, <hmnews at proconx.com> wrote:
>
>> Hi to all who responded,
>>
>> William's pull request reduces the total build time from 12s down to
>> 3.5s! So this would solve the issue.
>>
>> I compared some Scons versions for our reference project (C++, 161 C/C++
>> files eg targets) which is a real-world gcc cross-compilation project not
>> some odd test case.
>>
>> 2.3.0: 2.4s
>> 2.3.5: 3.15s
>> 2.3.6: 3.3s
>> 2.5.0: 12s
>> 2.5.0 with pull request applied: 3.5s
>>
>> Based on that, 2.3.0 is still the unbeaten winner, 2.3.5 and 2.3.6 I can
>> live with and once the pull request is applied to 2.5 build times are at
>> least again within a reasonable range for the project size.
>>
>> And yes, the build time scales with the number of object files in the
>> project.
>>
>> I can't test on Linux, my build environment lives purely on Windows.
>>
>> Thanks for all the help. Scons is a very useful tool.
>>
>> Henrik
>>
>>
>> William Blevins wrote:
>>
>>> Hi,
>>>
>>> This may be a known issue. SCons 2.5.0 changed how implicit dependencies
>>> were generated which added cross-language support (and fixed potentially
>>> missing implicit dependencies). As Alexey Klimkin has already pointed
>>> out, the change could be optimized safely by caching the path_func
>>> results. This wasn't something the original Executor code required,
>>> since scanner_key(s) never changed after previously picking the initial
>>> scanner. Please see
>>> https://bitbucket.org/scons/scons/pull-requests/345/optimize
>>> -implicit-dependency-scan/diff
>>>
>>> Your performance issue is most likely related. My fault entirely. I
>>> hadn't expected those calls to be that expensive or perhaps I just
>>> didn't have a test build big enough to notice.
>>>
>>> If you have the ability to try a patched version of SCons from the PR
>>> above, then feedback would be appreciated.
>>>
>>> V/R,
>>> William
>>>
>>> On Mon, Aug 8, 2016 at 9:37 AM, Vasily <just.one.man at yandex.ru
>>> <mailto:just.one.man at yandex.ru>> wrote:
>>>
>>>     Hi,
>>>
>>>     You might also do some profiling of fast and slow cases and compare
>>>     the results to see if there is any hint on what's slowing down your
>>>     build.
>>>
>>>     Thanks,
>>>     Vasily
>>>
>>>     08 авг. 2016 г. 10:17 пользователь "Dirk Bächle" <tshortik at gmx.de
>>>     <mailto:tshortik at gmx.de>> написал:
>>>
>>>         Hi Henrik,
>>>
>>>         On 08.08.2016 03:23, hmnews at proconx.com
>>>         <mailto:hmnews at proconx.com> wrote:
>>>
>>>             Hi all,
>>>
>>>             We just updated to Scons 2.5.0 after using happily 2.3.0 for
>>>             years.
>>>
>>>             Sadly we found performance to have dropped significantly and
>>>             I wonder
>>>             what the contributing factor could be.
>>>
>>>             Platform: Windows 10
>>>             Python: 2.7.12 64-bit
>>>
>>>
>>>         some more questions about this: Can you compare the times for a
>>>         build under Linux as well, or do you build under Windows only?
>>>
>>>             [...]
>>>
>>>             However with 2.5.0 execution time blows out to 11.8s, more
>>>             than 4 times
>>>             longer!!!
>>>
>>>
>>>         You say that this is a benchmark project. What happens to the
>>>         "real" projects? Do they scale, i.e. do they also take about 4
>>>         times longer each?
>>>
>>>             [...]
>>>
>>>             I appears that Scons takes much longer in 2.5.0 to analyse
>>>             the build
>>>             targets. This is also apparent when running scons-2.5.0
>>>             --debug=prepare
>>>             which takes noticeable longer to to print the file list.
>>>
>>>             I also compared the dependency tree for both version using
>>>             --debug=tree
>>>             and they are identical.
>>>
>>>
>>>         Good to know, because we have changed "scanning" in a few
>>>         places, which could've explained the longer times too.
>>>
>>>             Has anybody experienced a similar behaviour and is there any
>>>             tuning
>>>             parameter to make 2.5.0 offer the same performance of 2.3.0?
>>>
>>>
>>>         If there were such a parameter, it would definitely be "on" by
>>>         default. ;)
>>>
>>>
>>>             Is there any debug option to analyse why Scons 2.5.0 takes
>>>             such a long
>>>             execution time?
>>>
>>>
>>>         In general, SCons uses most of its time during a "zero build"
>>>         (nothing to do), "subst"ing variable values. If you're prepared
>>>         for a long and technical read have a look at
>>>         https://bitbucket.org/scons/scons/wiki/WhySconsIsNotSlow
>>>         <https://bitbucket.org/scons/scons/wiki/WhySconsIsNotSlow> . You
>>>         might also want to check out the "fastcpp" tool at
>>>         https://bitbucket.org/dirkbaechle/scons_fastcpp
>>>         <https://bitbucket.org/dirkbaechle/scons_fastcpp> to get an idea
>>>         of how to "tweak" your environment/toolchain to get more speed
>>>         in general.
>>>
>>>         I hope these first comments already help you a little further.
>>>
>>>         Best regards,
>>>
>>>         Dirk
>>>
>>>
>>>
>>>             Thanks
>>>
>>>             Henrik
>>>             _______________________________________________
>>>             Scons-users mailing list
>>>             Scons-users at scons.org <mailto:Scons-users at scons.org>
>>>             https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>             <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>>>
>>>
>>>         _______________________________________________
>>>         Scons-users mailing list
>>>         Scons-users at scons.org <mailto:Scons-users at scons.org>
>>>         https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>         <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>>>
>>>
>>>     _______________________________________________
>>>     Scons-users mailing list
>>>     Scons-users at scons.org <mailto:Scons-users at scons.org>
>>>     https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>     <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160809/e0a4cfdc/attachment.html>