[Scons-users] Parallel MD5

Eric van Gyzen eric at vangyzen.net
Wed Feb 26 11:55:44 EST 2014

On Feb 24, 2014, at 6:33 PM, Dirk Bächle wrote:

> On 22.02.2014 14:12, Eric van Gyzen wrote:

>> SCons Users:


>> It seems that SCons uses a single thread to calculate the MD5 sums of the sources, regardless of the -jN flag. Is this correct? When working in large projects with thousands of files, and only modifying one file, parallelizing the MD5 calculations across the -jN CPUs could be a major performance improvement.


> I've done some work on analysing and profiling SCons performance in the

> past (see http://www.scons.org/wiki/WhySconsIsNotSlow ), and so far I

> didn't find that MD5 computation is a real bottleneck.


> If you've experienced major slowdowns due to the calculation of

> hashsums, I'd be very interested in your results. This includes not only

> the actual figures, but also information about the used project setup

> and the evaluation process.


I am embarrassed to admit that my claim was based on a hunch, with no scientific experimentation of any kind. It is complete conjecture. Thank you for your performance analysis. It is very well done. Obviously, I need to do a similar analysis with our code base.

Kind regards,

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://four.pairlist.net/pipermail/scons-users/attachments/20140226/240ce26c/attachment.html

More information about the Scons-users mailing list