[Scons-users] Timing Issue with files retrieved from the SCons Cache

Fri Sep 30 12:55:29 EDT 2016

Shane,

This may be specific to Python file handling on Windows and not specific to
SCons 2.5.0. The mail archive is down currently, so I cannot link to the
full article, but here is an excerpt from one of the last conversations.

Hi All -
>

> Our resident Windows expert reached the following conclusions:
>
> Summary: It is a python bug in shutil.copy2
>
> The issue:
> When link.exe opens a obj file, it gets the following error:
> LINK : fatal error LNK1104: cannot open file 'build\cached\mongo\tools\mong
> obridge_options_init.obj'
>
> To diagnose these errors, I enabled ETW tracing with "FileIO stackwalk for
> FileCreate
> <https://msdn.microsoft.com/en-us/library/windows/desktop/aa964768%28v=vs.85%29.aspx>
> +FileCleanup+FileClose"
> <https://msdn.microsoft.com/en-us/library/windows/desktop/aa964773%28v=vs.85%29.aspx> and
> cranked through WPA with the data on Window 2008 R2 & Window 2012 R2. The
> 2012 R2 os offers stack traces which is why I used it
>
> By tracing the build, I can see that link.exe has a call to CreateFile
> fail with
> "A file cannot be opened because the share access flags are
> incompatible. (0xc0000043)"
>
> This occurs because it asked for a file with the following flags "file_open
> synchronous_io_nonalert non_directory_file shareRead", and another
> process had an existing handle to the file"file_overwrite_if
> synchronous_io_nonalert non_directory_file normal shareRead shareWrite".
>
> The existing process that had a handle to the file was none other then
> "python.exe" which created the file originally in copy2, but did not close
> it. I compared normal cases, and it does succesfully close the file. I do
> not know why like 1/100 or 1/200 times it fails.
>
> The workaround is win32file.CopyFile
> <http://timgolden.me.uk/python/win32_how_do_i/copy-a-file.html>.
>
> I patched FS.py with and it worked fine
> win32file.CopyFile(src, dst, 1)
>         return True
>
>
> I can confirm that the issue no-longer reproduces for me with the
> following change to FS.py:
>
> https://github.com/tychoish/mongo/commit/c8450fb4d304b2de06ba968b71f6ef
> acd3b5214e
>
> While I'd love to follow this deeper, debugging python's file system
> internals on Windows is not something I can really invest time in right
> now. We are most likely just going to make the above patch to our vendored
> copy of SCons and continue.
>
> Perhaps someone with more Python expertise would be interested in pursuing
> this further? I can give very detailed reproduction instructions.
>
> Thanks,
> Andrew
>

I do not think this has been officially patched in the development trunk
yet, but it is a known issue.

V/R,
William

On Fri, Sep 30, 2016 at 12:16 PM, Shane Gannon <sgannon200 at gmail.com> wrote:

> Hi
>
> I'm hitting a problem with the SCons Cache. Intermiddently (but too
> frequently) a build fails with an error like
>
> *Elapsed Time*
>
> *00:06:43.214* scons: building `out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj' because it doesn't exist*00:06:43.215* Retrieved `out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj' from cache
>
>
> *00:06:45.660* scons: building `src\jabber-client\jabber-build\Win32\bin\Release\memorymanager-unit-tests.exe' because it doesn't exist*00:06:45.661* LINK src\jabber-client\jabber-build\Win32\bin\Release\memorymanager-unit-tests.exe
>
> *00:06:45.694* LINK : fatal error LNK1104: cannot open file 'out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj'
>
>
> It's not specific to .obj files. It can happen for generated .h files,
> .libs, etc. Basically any file retrievable from the cache.
>
> This only occurs on a new build we're working to support. Namely an
> upgrade to Visual Studio 2015 where SCons 2.5.0 is used (versus 2.3.4 on
> the other machines).
>
> I noticed in the Release Notes the following entries for 2.5.0.
>
>     - SCons handles cache directories a bit differently/
>       - Cache files are now stored in 256 subdirectories in the cache directory by
>         default (this stresses NFS less). Existing cache directories will remain as
>         current, but SCons will prompt you to run scons-configure-cache which will
>         allow you to migrate to the new layout, or confirm you want to use the
>         existing layout.
>
>     - New external tool scons-configurecache which allows some configuration of
>       how files in the cache are controlled.
>
> I assume this might be causing the problem. Has anyone else reported an
> issue?
>
> Does anyone have a suggestion on how I can resolve or diagnois it?
>
> PS: We use Windows 10 on our build machines.
>
> Regards
> Shane
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160930/1338c835/attachment.html>