[Scons-users] empty files in cache directory after computer crash

[William Blevins]

Johannes,

That looks like good advice. Can you prepare a pull request? It seems you
have already looked into the source code?

V/R,
William

On Thu, Sep 15, 2016 at 7:03 AM, Abt, Johannes <johannes.abt at advantest.com>
wrote:

> Hi,
>
> I'm using the cache feature in SCons to speed up repeated builds. During
> the last days, some builds have been consistently failing.
>
> The reason seems to be that there are about 20 empty files in two of the
> cache directories. Both of the cache directories are on workstations that
> crashed and were restarted just before the build problems started.
>
> The cache directories are on local hard disks, on XFS filesystems. When
> googling for  journaling filesystems and empty files after crashes, that
> this problem is not uncommon.
>
> http://www.pointsoftware.ch/de/4-ext4-vs-ext3-filesystem-
> and-why-delayed-allocation-is-bad/ says that many applications rely on
> the following procedure.
>
> 1. open file for writing with a temporary name
> 2. write content
> 3. close file
> 4. rename file to final file name
>
> Most application developers think that all filesystems execute these
> operations in the same order. The page also says that this assumption is
> not true.
>
> In other words: After a crash, you can end up in a situation with a file
> having the final file name but only a part of of the content (or no content
> at all) because the file system write the  changes of step (4) before the
> content (2).
>
> Pointsoftware continues: "Good applications like PostgreSQL will not rely
> on default settings that are filesystem dependent and will trigger
> themselves a call to ‘fsync()'." (This would have to be done between step
> (2) and (3)).
>
> I have verified that SCons v2.3.3, when creating a cache entry, uses the
> approach of doing steps (1) to (4). However, SCons uses neither fsync() nor
> fdatasync() nor any other kind of sync. The same is true for SCons v2.5.0.
> I've used strace -f scons.. to check that.
>
> I recommend adding fdatasync() (or fsync()) when before closing the
> temporary cache file.
> And maybe there are other places in SCons where data integrity can be
> improved after a crash by adding f[data}sync().
>
> Regards
> Johannes Abt
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160915/4b5a4e95/attachment.html>