[Scons-users] Timing Issue with files retrieved from the SCons Cache

Fri Sep 30 13:56:10 EDT 2016

Hi William

Sounds like this issue could be related. But I think SCons 2.5.0 may be
playing it's part. The 2.3.4 build we have does not hit this problem.
Well... at least not with the same level of consistency.

Regards
Shane

On Fri, Sep 30, 2016 at 5:55 PM, William Blevins <wblevins001 at gmail.com>
wrote:

> Shane,
>
> This may be specific to Python file handling on Windows and not specific
> to SCons 2.5.0. The mail archive is down currently, so I cannot link to the
> full article, but here is an excerpt from one of the last conversations.
>
>
> Hi All -
>>
>
>> Our resident Windows expert reached the following conclusions:
>>
>> Summary: It is a python bug in shutil.copy2
>>
>> The issue:
>> When link.exe opens a obj file, it gets the following error:
>> LINK : fatal error LNK1104: cannot open file
>> 'build\cached\mongo\tools\mongobridge_options_init.obj'
>>
>> To diagnose these errors, I enabled ETW tracing with "FileIO stackwalk
>> for FileCreate
>> <https://msdn.microsoft.com/en-us/library/windows/desktop/aa964768%28v=vs.85%29.aspx>
>> +FileCleanup+FileClose"
>> <https://msdn.microsoft.com/en-us/library/windows/desktop/aa964773%28v=vs.85%29.aspx> and
>> cranked through WPA with the data on Window 2008 R2 & Window 2012 R2. The
>> 2012 R2 os offers stack traces which is why I used it
>>
>> By tracing the build, I can see that link.exe has a call to CreateFile
>> fail with
>> "A file cannot be opened because the share access flags are
>> incompatible. (0xc0000043)"
>>
>> This occurs because it asked for a file with the following flags "file_open
>> synchronous_io_nonalert non_directory_file shareRead", and another
>> process had an existing handle to the file"file_overwrite_if
>> synchronous_io_nonalert non_directory_file normal shareRead shareWrite".
>>
>> The existing process that had a handle to the file was none other then
>> "python.exe" which created the file originally in copy2, but did not close
>> it. I compared normal cases, and it does succesfully close the file. I do
>> not know why like 1/100 or 1/200 times it fails.
>>
>> The workaround is win32file.CopyFile
>> <http://timgolden.me.uk/python/win32_how_do_i/copy-a-file.html>.
>>
>> I patched FS.py with and it worked fine
>> win32file.CopyFile(src, dst, 1)
>>         return True
>>
>>
>> I can confirm that the issue no-longer reproduces for me with the
>> following change to FS.py:
>>
>> https://github.com/tychoish/mongo/commit/c8450fb4d304b2de06b
>> a968b71f6efacd3b5214e
>>
>> While I'd love to follow this deeper, debugging python's file system
>> internals on Windows is not something I can really invest time in right
>> now. We are most likely just going to make the above patch to our vendored
>> copy of SCons and continue.
>>
>> Perhaps someone with more Python expertise would be interested in
>> pursuing this further? I can give very detailed reproduction instructions.
>>
>> Thanks,
>> Andrew
>>
>
> I do not think this has been officially patched in the development trunk
> yet, but it is a known issue.
>
> V/R,
> William
>
>
> On Fri, Sep 30, 2016 at 12:16 PM, Shane Gannon <sgannon200 at gmail.com>
> wrote:
>
>> Hi
>>
>> I'm hitting a problem with the SCons Cache. Intermiddently (but too
>> frequently) a build fails with an error like
>>
>> *Elapsed Time*
>>
>> *00:06:43.214* scons: building `out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj' because it doesn't exist*00:06:43.215* Retrieved `out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj' from cache
>>
>>
>> *00:06:45.660* scons: building `src\jabber-client\jabber-build\Win32\bin\Release\memorymanager-unit-tests.exe' because it doesn't exist*00:06:45.661* LINK src\jabber-client\jabber-build\Win32\bin\Release\memorymanager-unit-tests.exe
>>
>> *00:06:45.694* LINK : fatal error LNK1104: cannot open file 'out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj'
>>
>>
>> It's not specific to .obj files. It can happen for generated .h files,
>> .libs, etc. Basically any file retrievable from the cache.
>>
>> This only occurs on a new build we're working to support. Namely an
>> upgrade to Visual Studio 2015 where SCons 2.5.0 is used (versus 2.3.4 on
>> the other machines).
>>
>> I noticed in the Release Notes the following entries for 2.5.0.
>>
>>     - SCons handles cache directories a bit differently/
>>       - Cache files are now stored in 256 subdirectories in the cache directory by
>>         default (this stresses NFS less). Existing cache directories will remain as
>>         current, but SCons will prompt you to run scons-configure-cache which will
>>         allow you to migrate to the new layout, or confirm you want to use the
>>         existing layout.
>>
>>     - New external tool scons-configurecache which allows some configuration of
>>       how files in the cache are controlled.
>>
>> I assume this might be causing the problem. Has anyone else reported an
>> issue?
>>
>> Does anyone have a suggestion on how I can resolve or diagnois it?
>>
>> PS: We use Windows 10 on our build machines.
>>
>> Regards
>> Shane
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160930/89cc383b/attachment-0001.html>