[Scons-users] Performance of copying from cache

Tom Tanner (BLOOMBERG/ LONDON) ttanner2 at bloomberg.net
Wed Jun 12 07:16:17 EDT 2013


So, we have a biggish build, which we tend to run with -j 2 or 4, with some large files on it and we've noticed that copying files out of our NFS cache can take ridiculous amounts of time (and we're talking about 10+ minutes, though AIX seems to be a lot worse than solaris or linux)

Having a hunt round, I discovered that shutil.copy2 copies 16k at a time, which doesn't seem terrifically efficient (it would appear I'm not the only person who thinks that). So I took a copy of that, used a 1M buffer, and it reduced my worst case copy to 11 seconds.

But the thing that really improved worst performance was replacing that with "cp -p". However, that hosed overall performance.

Has anyone any suggestions? Would it be saner to just read the while file at once (although as some of the files are quite large, that might be painful).


More information about the Scons-users mailing list