[Scons-users] empty files in cache directory after computer crash

[Abt, Johannes]

Hi,

I'm using the cache feature in SCons to speed up repeated builds. During the last days, some builds have been consistently failing. 

The reason seems to be that there are about 20 empty files in two of the cache directories. Both of the cache directories are on workstations that crashed and were restarted just before the build problems started.

The cache directories are on local hard disks, on XFS filesystems. When googling for  journaling filesystems and empty files after crashes, that this problem is not uncommon.

http://www.pointsoftware.ch/de/4-ext4-vs-ext3-filesystem-and-why-delayed-allocation-is-bad/ says that many applications rely on the following procedure.

1. open file for writing with a temporary name
2. write content
3. close file
4. rename file to final file name 

Most application developers think that all filesystems execute these operations in the same order. The page also says that this assumption is not true.

In other words: After a crash, you can end up in a situation with a file having the final file name but only a part of of the content (or no content at all) because the file system write the  changes of step (4) before the content (2).

Pointsoftware continues: "Good applications like PostgreSQL will not rely on default settings that are filesystem dependent and will trigger themselves a call to ‘fsync()'." (This would have to be done between step (2) and (3)).

I have verified that SCons v2.3.3, when creating a cache entry, uses the approach of doing steps (1) to (4). However, SCons uses neither fsync() nor fdatasync() nor any other kind of sync. The same is true for SCons v2.5.0. I've used strace -f scons.. to check that.

I recommend adding fdatasync() (or fsync()) when before closing the temporary cache file.
And maybe there are other places in SCons where data integrity can be improved after a crash by adding f[data}sync().

Regards
Johannes Abt