[Scons-users] Performance of copying from cache

[Tom Tanner (BLOOMBERG/ LONDON)]

So, we have a biggish build, which we tend to run with -j 2 or 4, with some large files on it and we've noticed that copying files out of our NFS cache can take ridiculous amounts of time (and we're talking about 10+ minutes, though AIX seems to be a lot worse than solaris or linux)

Having a hunt round, I discovered that shutil.copy2 copies 16k at a time, which doesn't seem terrifically efficient (it would appear I'm not the only person who thinks that). So I took a copy of that, used a 1M buffer, and it reduced my worst case copy to 11 seconds.

But the thing that really improved worst performance was replacing that with "cp -p". However, that hosed overall performance.

Has anyone any suggestions? Would it be saner to just read the while file at once (although as some of the files are quite large, that might be painful).