[Scons-users] Timing Issue with files retrieved from the SCons Cache

Fri Sep 30 16:25:34 EDT 2016

If we get enough good patches and aren't quite ready for 3, then there'll
be a 2.5.1 or 2.6.0 depending on what the changes are.

SCons's not quite py3 ready yet.

On Fri, Sep 30, 2016 at 4:17 PM, Shane Gannon <sgannon200 at gmail.com> wrote:

> That's be interesting. I thought the next release was going to be the
> Python 3 (but still supporting 2.7) version?
>
> On Fri, Sep 30, 2016 at 8:24 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
>
>> Welcome.
>>
>> We have several good updates since 2.5.0 available, so I am hoping that
>> we can get a few more in (like this issue) and do another release.
>>
>> V/R,
>> William
>>
>> On Fri, Sep 30, 2016 at 2:58 PM, Shane Gannon <sgannon200 at gmail.com>
>> wrote:
>>
>>> That makes sense. It also means a downgrade should be a viable option.
>>>
>>> Thanks William
>>>
>>> On Fri, Sep 30, 2016 at 7:11 PM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>>> Shane,
>>>>
>>>> Since this is a race condition, then that is expected. SCons 2.5.0 was
>>>> a major feature release. It added support for cross-language dependencies
>>>> which would affect timings on how long files may be open.
>>>>
>>>> V/R,
>>>> William
>>>>
>>>> On Fri, Sep 30, 2016 at 1:56 PM, Shane Gannon <sgannon200 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi William
>>>>>
>>>>> Sounds like this issue could be related. But I think SCons 2.5.0 may
>>>>> be playing it's part. The 2.3.4 build we have does not hit this problem.
>>>>> Well... at least not with the same level of consistency.
>>>>>
>>>>> Regards
>>>>> Shane
>>>>>
>>>>> On Fri, Sep 30, 2016 at 5:55 PM, William Blevins <
>>>>> wblevins001 at gmail.com> wrote:
>>>>>
>>>>>> Shane,
>>>>>>
>>>>>> This may be specific to Python file handling on Windows and not
>>>>>> specific to SCons 2.5.0. The mail archive is down currently, so I cannot
>>>>>> link to the full article, but here is an excerpt from one of the last
>>>>>> conversations.
>>>>>>
>>>>>>
>>>>>> Hi All -
>>>>>>>
>>>>>>
>>>>>>> Our resident Windows expert reached the following conclusions:
>>>>>>>
>>>>>>> Summary: It is a python bug in shutil.copy2
>>>>>>>
>>>>>>> The issue:
>>>>>>> When link.exe opens a obj file, it gets the following error:
>>>>>>> LINK : fatal error LNK1104: cannot open file
>>>>>>> 'build\cached\mongo\tools\mongobridge_options_init.obj'
>>>>>>>
>>>>>>> To diagnose these errors, I enabled ETW tracing with "FileIO
>>>>>>> stackwalk for FileCreate
>>>>>>> <https://msdn.microsoft.com/en-us/library/windows/desktop/aa964768%28v=vs.85%29.aspx>
>>>>>>> +FileCleanup+FileClose"
>>>>>>> <https://msdn.microsoft.com/en-us/library/windows/desktop/aa964773%28v=vs.85%29.aspx> and
>>>>>>> cranked through WPA with the data on Window 2008 R2 & Window 2012 R2. The
>>>>>>> 2012 R2 os offers stack traces which is why I used it
>>>>>>>
>>>>>>> By tracing the build, I can see that link.exe has a call to
>>>>>>> CreateFile fail with
>>>>>>> "A file cannot be opened because the share access flags are
>>>>>>> incompatible. (0xc0000043)"
>>>>>>>
>>>>>>> This occurs because it asked for a file with the following flags "file_open
>>>>>>> synchronous_io_nonalert non_directory_file shareRead", and another
>>>>>>> process had an existing handle to the file"file_overwrite_if
>>>>>>> synchronous_io_nonalert non_directory_file normal shareRead shareWrite".
>>>>>>>
>>>>>>> The existing process that had a handle to the file was none other
>>>>>>> then "python.exe" which created the file originally in copy2, but did not
>>>>>>> close it. I compared normal cases, and it does succesfully close the file.
>>>>>>> I do not know why like 1/100 or 1/200 times it fails.
>>>>>>>
>>>>>>> The workaround is win32file.CopyFile
>>>>>>> <http://timgolden.me.uk/python/win32_how_do_i/copy-a-file.html>.
>>>>>>>
>>>>>>> I patched FS.py with and it worked fine
>>>>>>> win32file.CopyFile(src, dst, 1)
>>>>>>>         return True
>>>>>>>
>>>>>>>
>>>>>>> I can confirm that the issue no-longer reproduces for me with the
>>>>>>> following change to FS.py:
>>>>>>>
>>>>>>> https://github.com/tychoish/mongo/commit/c8450fb4d304b2de06b
>>>>>>> a968b71f6efacd3b5214e
>>>>>>>
>>>>>>> While I'd love to follow this deeper, debugging python's file system
>>>>>>> internals on Windows is not something I can really invest time in right
>>>>>>> now. We are most likely just going to make the above patch to our vendored
>>>>>>> copy of SCons and continue.
>>>>>>>
>>>>>>> Perhaps someone with more Python expertise would be interested in
>>>>>>> pursuing this further? I can give very detailed reproduction instructions.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>>
>>>>>>
>>>>>> I do not think this has been officially patched in the development
>>>>>> trunk yet, but it is a known issue.
>>>>>>
>>>>>> V/R,
>>>>>> William
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 30, 2016 at 12:16 PM, Shane Gannon <sgannon200 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I'm hitting a problem with the SCons Cache. Intermiddently (but too
>>>>>>> frequently) a build fails with an error like
>>>>>>>
>>>>>>> *Elapsed Time*
>>>>>>>
>>>>>>> *00:06:43.214* scons: building `out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj' because it doesn't exist*00:06:43.215* Retrieved `out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj' from cache
>>>>>>>
>>>>>>>
>>>>>>> *00:06:45.660* scons: building `src\jabber-client\jabber-build\Win32\bin\Release\memorymanager-unit-tests.exe' because it doesn't exist*00:06:45.661* LINK src\jabber-client\jabber-build\Win32\bin\Release\memorymanager-unit-tests.exe
>>>>>>>
>>>>>>> *00:06:45.694* LINK : fatal error LNK1104: cannot open file 'out\windows-x86-MD-unicode-vs2015-rel\obj-static\components\memorymanager\tests\unit_tests\TestMemoryManager.obj'
>>>>>>>
>>>>>>>
>>>>>>> It's not specific to .obj files. It can happen for generated .h
>>>>>>> files, .libs, etc. Basically any file retrievable from the cache.
>>>>>>>
>>>>>>> This only occurs on a new build we're working to support. Namely an
>>>>>>> upgrade to Visual Studio 2015 where SCons 2.5.0 is used (versus 2.3.4 on
>>>>>>> the other machines).
>>>>>>>
>>>>>>> I noticed in the Release Notes the following entries for 2.5.0.
>>>>>>>
>>>>>>>     - SCons handles cache directories a bit differently/
>>>>>>>       - Cache files are now stored in 256 subdirectories in the cache directory by
>>>>>>>         default (this stresses NFS less). Existing cache directories will remain as
>>>>>>>         current, but SCons will prompt you to run scons-configure-cache which will
>>>>>>>         allow you to migrate to the new layout, or confirm you want to use the
>>>>>>>         existing layout.
>>>>>>>
>>>>>>>     - New external tool scons-configurecache which allows some configuration of
>>>>>>>       how files in the cache are controlled.
>>>>>>>
>>>>>>> I assume this might be causing the problem. Has anyone else reported
>>>>>>> an issue?
>>>>>>>
>>>>>>> Does anyone have a suggestion on how I can resolve or diagnois it?
>>>>>>>
>>>>>>> PS: We use Windows 10 on our build machines.
>>>>>>>
>>>>>>> Regards
>>>>>>> Shane
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160930/02424e31/attachment-0001.html>