[Scons-users] Timing Issue with files retrieved from the SCons Cache

Fri Sep 30 15:24:45 EDT 2016

Welcome.

We have several good updates since 2.5.0 available, so I am hoping that we
can get a few more in (like this issue) and do another release.

V/R,
William

On Fri, Sep 30, 2016 at 2:58 PM, Shane Gannon <sgannon200 at gmail.com> wrote:

> That makes sense. It also means a downgrade should be a viable option.
>
> Thanks William
>
> On Fri, Sep 30, 2016 at 7:11 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
>
>> Shane,
>>
>> Since this is a race condition, then that is expected. SCons 2.5.0 was a
>> major feature release. It added support for cross-language dependencies
>> which would affect timings on how long files may be open.
>>
>> V/R,
>> William
>>
>> On Fri, Sep 30, 2016 at 1:56 PM, Shane Gannon <sgannon200 at gmail.com>
>> wrote:
>>
>>> Hi William
>>>
>>> Sounds like this issue could be related. But I think SCons 2.5.0 may be
>>> playing it's part. The 2.3.4 build we have does not hit this problem.
>>> Well... at least not with the same level of consistency.
>>>
>>> Regards
>>> Shane
>>>
>>> On Fri, Sep 30, 2016 at 5:55 PM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>>> Shane,
>>>>
>>>> This may be specific to Python file handling on Windows and not
>>>> specific to SCons 2.5.0. The mail archive is down currently, so I cannot
>>>> link to the full article, but here is an excerpt from one of the last
>>>> conversations.
>>>>
>>>>
>>>> Hi All -
>>>>>
>>>>
>>>>> Our resident Windows expert reached the following conclusions:
>>>>>
>>>>> Summary: It is a python bug in shutil.copy2
>>>>>
>>>>> The issue:
>>>>> When link.exe opens a obj file, it gets the following error:
>>>>> LINK : fatal error LNK1104: cannot open file
>>>>> 'build\cached\mongo\tools\mongobridge_options_init.obj'
>>>>>
>>>>> To diagnose these errors, I enabled ETW tracing with "FileIO stackwalk
>>>>> for FileCreate
>>>>> <https://msdn.microsoft.com/en-us/library/windows/desktop/aa964768%28v=vs.85%29.aspx>
>>>>> +FileCleanup+FileClose"
>>>>> <https://msdn.microsoft.com/en-us/library/windows/desktop/aa964773%28v=vs.85%29.aspx> and
>>>>> cranked through WPA with the data on Window 2008 R2 & Window 2012 R2. The
>>>>> 2012 R2 os offers stack traces which is why I used it
>>>>>
>>>>> By tracing the build, I can see that link.exe has a call to CreateFile
>>>>> fail with
>>>>> "A file cannot be opened because the share access flags are
>>>>> incompatible. (0xc0000043)"
>>>>>
>>>>> This occurs because it asked for a file with the following flags "file_open
>>>>> synchronous_io_nonalert non_directory_file shareRead", and another
>>>>> process had an existing handle to the file"file_overwrite_if
>>>>> synchronous_io_nonalert non_directory_file normal shareRead shareWrite".
>>>>>
>>>>> The existing process that had a handle to the file was none other then
>>>>> "python.exe" which created the file originally in copy2, but did not close
>>>>> it. I compared normal cases, and it does succesfully close the file. I do
>>>>> not know why like 1/100 or 1/200 times it fails.
>>>>>
>>>>> The workaround is win32file.CopyFile
>>>>> <http://timgolden.me.uk/python/win32_how_do_i/copy-a-file.html>.
>>>>>
>>>>> I patched FS.py with and it worked fine
>>>>> win32file.CopyFile(src, dst, 1)
>>>>>         return True
>>>>>
>>>>>
>>>>> I can confirm that the issue no-longer reproduces for me with the
>>>>> following change to FS.py:
>>>>>
>>>>> https://github.com/tychoish/mongo/commit/c8450fb4d304b2de06b
>>>>> a968b71f6efacd3b5214e
>>>>>
>>>>> While I'd love to follow this deeper, debugging python's file system
>>>>> internals on Windows is not something I can really invest time in right
>>>>> now. We are most likely just going to make the above patch to our vendored
>>>>> copy of SCons and continue.
>>>>>
>>>>> Perhaps someone with more Python expertise would be interested in
>>>>> pursuing this further? I can give very detailed reproduction instructions.
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>
>>>> I do not think this has been officially patched in the development
>>>> trunk yet, but it is a known issue.
>>>>
>>>> V/R,
>>>> William
>>>>
>>>>
>>>> On Fri, Sep 30, 2016 at 12:16 PM, Shane Gannon <sgannon200 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I'm hitting a problem with the SCons Cache. Intermiddently (but too
>>>>> frequently) a build fails with an error like
>>>>>
>>>>> *Elapsed Time*
>>>>>
>>>>> *00:06:43.214* scons: building `out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj' because it doesn't exist*00:06:43.215* Retrieved `out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj' from cache
>>>>>
>>>>>
>>>>> *00:06:45.660* scons: building `src\jabber-client\jabber-build\Win32\bin\Release\memorymanager-unit-tests.exe' because it doesn't exist*00:06:45.661* LINK src\jabber-client\jabber-build\Win32\bin\Release\memorymanager-unit-tests.exe
>>>>>
>>>>> *00:06:45.694* LINK : fatal error LNK1104: cannot open file 'out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj'
>>>>>
>>>>>
>>>>> It's not specific to .obj files. It can happen for generated .h files,
>>>>> .libs, etc. Basically any file retrievable from the cache.
>>>>>
>>>>> This only occurs on a new build we're working to support. Namely an
>>>>> upgrade to Visual Studio 2015 where SCons 2.5.0 is used (versus 2.3.4 on
>>>>> the other machines).
>>>>>
>>>>> I noticed in the Release Notes the following entries for 2.5.0.
>>>>>
>>>>>     - SCons handles cache directories a bit differently/
>>>>>       - Cache files are now stored in 256 subdirectories in the cache directory by
>>>>>         default (this stresses NFS less). Existing cache directories will remain as
>>>>>         current, but SCons will prompt you to run scons-configure-cache which will
>>>>>         allow you to migrate to the new layout, or confirm you want to use the
>>>>>         existing layout.
>>>>>
>>>>>     - New external tool scons-configurecache which allows some configuration of
>>>>>       how files in the cache are controlled.
>>>>>
>>>>> I assume this might be causing the problem. Has anyone else reported
>>>>> an issue?
>>>>>
>>>>> Does anyone have a suggestion on how I can resolve or diagnois it?
>>>>>
>>>>> PS: We use Windows 10 on our build machines.
>>>>>
>>>>> Regards
>>>>> Shane
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160930/723b51d0/attachment.html>