[Scons-users] Possible 2.5.1 regression on Windows? (was: Unreliable build problem)
Hill, Steve (FP COM)
Steve.Hill at cobham.com
Fri May 5 08:46:49 EDT 2017
That's interesting. Since you found the need to do that, I decided to try a much simpler patch^H^H^H^H^H hack to see whether it had an effect:
def another_open(name, mode="r", *args, **kwds):
flags = os.O_NOINHERIT
read = "r" in mode
write = "w" in mode
append = "a" in mode
rw = "+" in mode
binary = "b" in mode
if not read and not write and not append:
read = True
flags |= os.O_BINARY if binary else os.O_TEXT
if not read:
flags |= os.O_CREAT
if write:
flags |= os.O_TRUNC
elif append:
flags |= os.O_APPEND
flags |= os.O_RDWR if rw else os.O_RDONLY if read else os.O_WRONLY
fd = os.open(name, flags)
return os.fdopen(fd, mode, *args, **kwds)
import __builtin__
__builtin__.open = another_open
__builtin__.file = another_open
I've performed a number of builds (each producing around 700 executables) and it hasn't failed once. When I removed the patch, the first build failed with permission denied.
I've no idea why this has made a difference, since I assume that it is basically doing the same thing as _scons_open and scons_file (once the lock was put in place). I'll try to get one of the guinea-pigs that are trying 2.5.1 to put this in place and see if it resolves the issue for them too...
S.
--
From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Jason Kenny
Sent: 04 May 2017 21:30
To: SCons users mailing list
Subject: Re: [Scons-users] Possible 2.5.1 regression on Windows? (was: Unreliable build problem)
I am coming in late on this.. but the code in question always broke my windows builds
Plus is break tools the MS tools that use hardlink or symlinks behind the covers to speed up work.
The way I got around this was this file I have in Parts. It a bigger monkey patch to the whole file code in python to allow it open files with the correct base permissions. It bypasses the c like fopen() problems. Since Parts uses a function that calls subprocess for the SPAWN logic this should work as well.
Anyways this is the file in Parts. It is setup to be a general easy drop in file to SCons
https://bitbucket.org/sconsparts/parts/src/bbe2af951c83e976782930bb8ec9f5466560286b/parts/overrides/os_file.py?at=master&fileviewer=file-view-default
Jason
From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Bill Deegan
Sent: Thursday, May 4, 2017 10:15 AM
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] Possible 2.5.1 regression on Windows? (was: Unreliable build problem)
Steve,
Can you try something since you've got a pretty reproducible configuration.
Add this to your top level SConstruct
sys.setcheckinterval(1000)
(This changes the thread swap checking interval, default on py2.7 is 100)
Thanks,
Bill
On Thu, May 4, 2017 at 12:02 AM, William Blevins <wblevins001 at gmail.com> wrote:
Fair enough. I was just remembering the hot topic of race conditions
on Windows and trying to drudge up some recent information ;)
On Wed, May 3, 2017 at 9:53 PM, Bill Deegan <bill at baddogconsulting.com> wrote:
> Nope.. actually it's all 3.
>
> The issue (at it's root) is likely either threads are sharing the handle and
> it's not closed in a timely fashion on windows, or just plain normal file
> closes aren't being closed before the call returns on win32.
> Of note, looking at the c code, the GIL is released around the fclose()..
>
> -Bill
>
> On Wed, May 3, 2017 at 9:09 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
>>
>> Bill,
>>
>> I think you mean:
>> http://scons.tigris.org/issues/show_bug.cgi?id=2449
>>
>> https://bitbucket.org/scons/scons/pull-requests/389/fix-race-condition-on-win32/diff
>>
>> V/R,
>> William
>>
>> On Wed, May 3, 2017 at 6:32 PM, Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>> > Likely it's the same issue as this:
>> > http://scons.tigris.org/issues/show_bug.cgi?id=2124
>> >
>> >
>> >
>> > On Wed, May 3, 2017 at 3:14 PM, Arvid Rosén <arvid at softube.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I have problems with that too. Typically running 16 threads. I had to
>> >> rewrite all actions that generated source files (like header files with
>> >> version information), because they never worked reliably on Windows.
>> >> Mac has
>> >> always been fine. I ended up using static header files with defines
>> >> passed
>> >> on the command line instead. Somewhat ugly but it works.
>> >>
>> >> I always thought this was rather related to some Windows related
>> >> scanning
>> >> or anti-virus service, but I might give it a try with 2.3.6 and see if
>> >> that
>> >> solves it.
>> >>
>> >> Cheers,
>> >> Arvid
>> >>
>> >> Get Outlook for iOS
>> >> _____________________________
>> >> From: Bill Deegan <bill at baddogconsulting.com>
>> >> Sent: onsdag, maj 3, 2017 8:25 em
>> >> Subject: Re: [Scons-users] Possible 2.5.1 regression on Windows? (was:
>> >> Unreliable build problem)
>> >> To: SCons users mailing list <scons-users at scons.org>
>> >>
>> >>
>> >>
>> >> Steve,
>> >>
>> >> That's useful to know 2.3.6 isn't showing this.
>> >> We've had a few reports of others running into it.
>> >>
>> >> Are you using CacheDirs?
>> >>
>> >> -Bill
>> >>
>> >> On Wed, May 3, 2017 at 10:42 AM, Hill, Steve (FP COM)
>> >> <Steve.Hill at cobham.com> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> While looking at the unreliable build problem (which I will return
>> >>> to), I
>> >>> upgraded to 2.5.1 (from 2.3.6). Unfortunately, this seems to have
>> >>> introduced
>> >>> an issue which is preventing me from rolling the new version out the
>> >>> development community: I am seeing the file handle inheritance problem
>> >>> again
>> >>> that _scons_file and _scons_open were added to work around.
>> >>>
>> >>> We typically use between 8 and 12 build threads (depending on the
>> >>> physical machine that we are building on) and, when building with
>> >>> 2.5.1, we
>> >>> quite frequently see the build fail with some sort of access denied
>> >>> issue.
>> >>> The build system automatically runs handle.exe in this case and I can
>> >>> see
>> >>> that the other (unrelated) build threads have an open handle on the
>> >>> file at
>> >>> this time. Reverting to 2.3.6 results in the issue going away.
>> >>>
>> >>> I've confirmed that _scons_open and _scons_file are both in place for
>> >>> the
>> >>> built-in file() and open() and I've monkey patched os.open to assert
>> >>> that it
>> >>> is always called with the os.O_NOINHERIT flag. Does anyone know what
>> >>> other
>> >>> functions could be causing this that I can check?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> S.
>> >>>
>> >>> _______________________________________________
>> >>> Scons-users mailing list
>> >>> Scons-users at scons.org
>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>>
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> Scons-users mailing list
>> >> Scons-users at scons.org
>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>
>> >
>> >
>> > _______________________________________________
>> > Scons-users mailing list
>> > Scons-users at scons.org
>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 526 bytes
Desc: not available
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170505/371640db/attachment.pgp>
More information about the Scons-users
mailing list