[Scons-users] Possible 2.5.1 regression on Windows? (was: Unreliable build problem)

Thu May 4 16:30:15 EDT 2017

I am coming in late on this.. but the code in question always broke my windows builds

Plus is break tools the MS tools that use hardlink or symlinks behind the covers to speed up work.

The way I got around this was this file I have in Parts. It a bigger monkey patch to the whole file code in python to allow it open files with the correct base permissions. It bypasses the c like fopen() problems. Since Parts uses a function that calls subprocess for the SPAWN logic this should work as well.

Anyways this is the file in Parts. It is setup to be a general easy drop in file to SCons

https://bitbucket.org/sconsparts/parts/src/bbe2af951c83e976782930bb8ec9f5466560286b/parts/overrides/os_file.py?at=master&fileviewer=file-view-default

Jason

From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of Bill Deegan
Sent: Thursday, May 4, 2017 10:15 AM
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] Possible 2.5.1 regression on Windows? (was: Unreliable build problem)

Steve,
Can you try something since you've got a pretty reproducible configuration.
Add this to your top level SConstruct

sys.setcheckinterval(1000)
(This changes the thread swap checking interval, default on py2.7 is 100)
Thanks,
Bill

On Thu, May 4, 2017 at 12:02 AM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:
Fair enough. I was just remembering the hot topic of race conditions
on Windows and trying to drudge up some recent information ;)

On Wed, May 3, 2017 at 9:53 PM, Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>> wrote:
> Nope.. actually it's all 3.
>
> The issue (at it's root) is likely either threads are sharing the handle and
> it's not closed in a timely fashion on windows, or just plain normal file
> closes aren't being closed before the call returns on win32.
> Of note, looking at the c code, the GIL is released around the fclose()..
>
> -Bill
>
> On Wed, May 3, 2017 at 9:09 PM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>>
> wrote:
>>
>> Bill,
>>
>> I think you mean:
>> http://scons.tigris.org/issues/show_bug.cgi?id=2449
>>
>> https://bitbucket.org/scons/scons/pull-requests/389/fix-race-condition-on-win32/diff
>>
>> V/R,
>> William
>>
>> On Wed, May 3, 2017 at 6:32 PM, Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>>
>> wrote:
>> > Likely it's the same issue as this:
>> > http://scons.tigris.org/issues/show_bug.cgi?id=2124
>> >
>> >
>> >
>> > On Wed, May 3, 2017 at 3:14 PM, Arvid Rosén <arvid at softube.com<mailto:arvid at softube.com>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I have problems with that too. Typically running 16 threads. I had to
>> >> rewrite all actions that generated source files (like header files with
>> >> version information), because they never worked reliably on Windows.
>> >> Mac has
>> >> always been fine. I ended up using static header files with defines
>> >> passed
>> >> on the command line instead. Somewhat ugly but it works.
>> >>
>> >> I always thought this was rather related to some Windows related
>> >> scanning
>> >> or anti-virus service, but I might give it a try with 2.3.6 and see if
>> >> that
>> >> solves it.
>> >>
>> >> Cheers,
>> >> Arvid
>> >>
>> >> Get Outlook for iOS
>> >> _____________________________
>> >> From: Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>>
>> >> Sent: onsdag, maj 3, 2017 8:25 em
>> >> Subject: Re: [Scons-users] Possible 2.5.1 regression on Windows? (was:
>> >> Unreliable build problem)
>> >> To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
>> >>
>> >>
>> >>
>> >> Steve,
>> >>
>> >> That's useful to know 2.3.6 isn't showing this.
>> >> We've had a few reports of others running into it.
>> >>
>> >> Are you using CacheDirs?
>> >>
>> >> -Bill
>> >>
>> >> On Wed, May 3, 2017 at 10:42 AM, Hill, Steve (FP COM)
>> >> <Steve.Hill at cobham.com<mailto:Steve.Hill at cobham.com>> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> While looking at the unreliable build problem (which I will return
>> >>> to), I
>> >>> upgraded to 2.5.1 (from 2.3.6). Unfortunately, this seems to have
>> >>> introduced
>> >>> an issue which is preventing me from rolling the new version out the
>> >>> development community: I am seeing the file handle inheritance problem
>> >>> again
>> >>> that _scons_file and _scons_open were added to work around.
>> >>>
>> >>> We typically use between 8 and 12 build threads (depending on the
>> >>> physical machine that we are building on) and, when building with
>> >>> 2.5.1, we
>> >>> quite frequently see the build fail with some sort of access denied
>> >>> issue.
>> >>> The build system automatically runs handle.exe in this case and I can
>> >>> see
>> >>> that the other (unrelated) build threads have an open handle on the
>> >>> file at
>> >>> this time. Reverting to 2.3.6 results in the issue going away.
>> >>>
>> >>> I've confirmed that _scons_open and _scons_file are both in place for
>> >>> the
>> >>> built-in file() and open() and I've monkey patched os.open to assert
>> >>> that it
>> >>> is always called with the os.O_NOINHERIT flag. Does anyone know what
>> >>> other
>> >>> functions could be causing this that I can check?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> S.
>> >>>
>> >>> _______________________________________________
>> >>> Scons-users mailing list
>> >>> Scons-users at scons.org<mailto:Scons-users at scons.org>
>> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>>
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> Scons-users mailing list
>> >> Scons-users at scons.org<mailto:Scons-users at scons.org>
>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >>
>> >
>> >
>> > _______________________________________________
>> > Scons-users mailing list
>> > Scons-users at scons.org<mailto:Scons-users at scons.org>
>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org<mailto:Scons-users at scons.org>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org<mailto:Scons-users at scons.org>
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170504/9829593c/attachment-0001.html>