[Scons-users] Performance of version 2.5.0 vs 2.3.0 on Windows host dropped significantly

Tue Aug 9 08:45:28 EDT 2016

Hi to all who responded,

William's pull request reduces the total build time from 12s down to 
3.5s! So this would solve the issue.

I compared some Scons versions for our reference project (C++, 161 C/C++ 
files eg targets) which is a real-world gcc cross-compilation project 
not some odd test case.

2.3.0: 2.4s
2.3.5: 3.15s
2.3.6: 3.3s
2.5.0: 12s
2.5.0 with pull request applied: 3.5s

Based on that, 2.3.0 is still the unbeaten winner, 2.3.5 and 2.3.6 I can 
live with and once the pull request is applied to 2.5 build times are at 
least again within a reasonable range for the project size.

And yes, the build time scales with the number of object files in the 
project.

I can't test on Linux, my build environment lives purely on Windows.

Thanks for all the help. Scons is a very useful tool.

Henrik

William Blevins wrote:
> Hi,
>
> This may be a known issue. SCons 2.5.0 changed how implicit dependencies
> were generated which added cross-language support (and fixed potentially
> missing implicit dependencies). As Alexey Klimkin has already pointed
> out, the change could be optimized safely by caching the path_func
> results. This wasn't something the original Executor code required,
> since scanner_key(s) never changed after previously picking the initial
> scanner. Please see
> https://bitbucket.org/scons/scons/pull-requests/345/optimize-implicit-dependency-scan/diff
>
> Your performance issue is most likely related. My fault entirely. I
> hadn't expected those calls to be that expensive or perhaps I just
> didn't have a test build big enough to notice.
>
> If you have the ability to try a patched version of SCons from the PR
> above, then feedback would be appreciated.
>
> V/R,
> William
>
> On Mon, Aug 8, 2016 at 9:37 AM, Vasily <just.one.man at yandex.ru
> <mailto:just.one.man at yandex.ru>> wrote:
>
>     Hi,
>
>     You might also do some profiling of fast and slow cases and compare
>     the results to see if there is any hint on what's slowing down your
>     build.
>
>     Thanks,
>     Vasily
>
>     08 авг. 2016 г. 10:17 пользователь "Dirk Bächle" <tshortik at gmx.de
>     <mailto:tshortik at gmx.de>> написал:
>
>         Hi Henrik,
>
>         On 08.08.2016 03:23, hmnews at proconx.com
>         <mailto:hmnews at proconx.com> wrote:
>
>             Hi all,
>
>             We just updated to Scons 2.5.0 after using happily 2.3.0 for
>             years.
>
>             Sadly we found performance to have dropped significantly and
>             I wonder
>             what the contributing factor could be.
>
>             Platform: Windows 10
>             Python: 2.7.12 64-bit
>
>
>         some more questions about this: Can you compare the times for a
>         build under Linux as well, or do you build under Windows only?
>
>             [...]
>
>             However with 2.5.0 execution time blows out to 11.8s, more
>             than 4 times
>             longer!!!
>
>
>         You say that this is a benchmark project. What happens to the
>         "real" projects? Do they scale, i.e. do they also take about 4
>         times longer each?
>
>             [...]
>
>             I appears that Scons takes much longer in 2.5.0 to analyse
>             the build
>             targets. This is also apparent when running scons-2.5.0
>             --debug=prepare
>             which takes noticeable longer to to print the file list.
>
>             I also compared the dependency tree for both version using
>             --debug=tree
>             and they are identical.
>
>
>         Good to know, because we have changed "scanning" in a few
>         places, which could've explained the longer times too.
>
>             Has anybody experienced a similar behaviour and is there any
>             tuning
>             parameter to make 2.5.0 offer the same performance of 2.3.0?
>
>
>         If there were such a parameter, it would definitely be "on" by
>         default. ;)
>
>
>             Is there any debug option to analyse why Scons 2.5.0 takes
>             such a long
>             execution time?
>
>
>         In general, SCons uses most of its time during a "zero build"
>         (nothing to do), "subst"ing variable values. If you're prepared
>         for a long and technical read have a look at
>         https://bitbucket.org/scons/scons/wiki/WhySconsIsNotSlow
>         <https://bitbucket.org/scons/scons/wiki/WhySconsIsNotSlow> . You
>         might also want to check out the "fastcpp" tool at
>         https://bitbucket.org/dirkbaechle/scons_fastcpp
>         <https://bitbucket.org/dirkbaechle/scons_fastcpp> to get an idea
>         of how to "tweak" your environment/toolchain to get more speed
>         in general.
>
>         I hope these first comments already help you a little further.
>
>         Best regards,
>
>         Dirk
>
>
>
>             Thanks
>
>             Henrik
>             _______________________________________________
>             Scons-users mailing list
>             Scons-users at scons.org <mailto:Scons-users at scons.org>
>             https://pairlist4.pair.net/mailman/listinfo/scons-users
>             <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>
>
>         _______________________________________________
>         Scons-users mailing list
>         Scons-users at scons.org <mailto:Scons-users at scons.org>
>         https://pairlist4.pair.net/mailman/listinfo/scons-users
>         <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>
>
>     _______________________________________________
>     Scons-users mailing list
>     Scons-users at scons.org <mailto:Scons-users at scons.org>
>     https://pairlist4.pair.net/mailman/listinfo/scons-users
>     <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users