[Scons-users] Intermittent Install() failure

Arvid Rosén arvid at softube.com
Tue Jul 19 16:48:48 EDT 2016


I have had problems with builders using Python open() in Builders on Windows. How should that be handled to be safe?

Cheers,
Arvid

Get Outlook for iOS<https://aka.ms/o0ukef>

_____________________________
From: William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>>
Sent: tisdag, juli 19, 2016 7:48 em
Subject: Re: [Scons-users] Intermittent Install() failure
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>


Bill,

I assume you think this may be a problem with not explicitly closing the file? Relying on python garbage collection in this case would certainly be dangerous.

V/R,
William

On Tue, Jul 19, 2016 at 6:22 PM, Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>> wrote:
Steve,

Do you have any python logic creating the files in the SCons process? (a builder where the builder actuall does open('file','w').. to create the file?

-Bill

On Tue, Jul 19, 2016 at 7:55 AM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:
Steve,

As a final thought, I have used SCons to build HT 24-core server blades (so 48 w/ HT) that RAM boot (no harddrive at RT). If it was specific to the core package, I imagine I would have seen this issue tons.

V/R,
William

On Tue, Jul 19, 2016 at 12:52 PM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:
Vasily,

I think you are referring to Precious.

V/R,
William

On Tue, Jul 19, 2016 at 9:51 AM, Vasily <just.one.man at yandex.ru<mailto:just.one.man at yandex.ru>> wrote:

Hi Steve,

What does your build do with this file except installing it? Is it used by a compiler or some other tool?

Also, if I'm not mistaken, default SCons behavior is to remove the target before performing any action to regenerate it, so this might be the source of the exception you're seeing. There seems to be a way to turn off the removal part of the action, so you may want to check the manual and see if it helps you.

P.S. Based on file path I assume you're working on Windows, is this path a network one?

Thanks,
Vasily

19 §Ú§ð§Ý§ñ 2016 §Ô. 11:02 §á§à§Ý§î§Ù§à§Ó§Ñ§ä§Ö§Ý§î "Hill, Steve (FP COM)" <Steve.Hill at cobham.com<mailto:Steve.Hill at cobham.com>> §ß§Ñ§á§Ú§ã§Ñ§Ý:

Thanks William.

I¡¯ve checked the dependency tree and, as far as I can tell, it looks OK.
We do have custom scanners ¨C using env.Scanner(_scan_domain_header) ¨C but not for the files that are affected.
As far as I can see, the fix for that Python bug never made it into 2.6.x but it appears always to result in a ¡°No child processes¡± OSError so I don¡¯t believe that is what I am seeing.

Note that I have (temporarily) changed the decider below to simply return True (without doing anything with the file) and I still see the issue so the decider doesn¡¯t seem to be relevant to the issue.

Does anyone have any more ideas? This is becoming a major issue for our automated builds¡­

S.


Steve,
I of course should have asked the obvious question, do you know if you dependency tree has missing dependencies? This tends to be a common issue.
V/R,
William

On Fri, Jul 15, 2016 at 5:14 PM, William Blevins <wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>> wrote:
Steve,
I'm not aware of any specific issue with install, but there are some possible issues that I am aware:

  1.  If you have custom scanners, make sure they implement from SCons.Scanner.Current and not SCons.Scanner.Base; otherwise, you might have concurrent file access between implicit scanning operations and other processes.
  2.  There was a big subprocess bug in python 2.6 that carried through several other major versions: https://bugs.python.org/issue1731717. I would check to see that your version of 2.6 contains the patch for this issue.

V/R,

William

On Fri, Jul 15, 2016 at 9:30 AM, Hill, Steve (FP COM) <Steve.Hill at cobham.com<mailto:Steve.Hill at cobham.com>> wrote:
I¡¯m having a problem where, with parallel builds (most people use 8, 12 or 16 threads), we occasionally get failures like the following:


F:\<directory path>\hw_cfgs\1Server_1TM_6C66_2U.cfg: The process cannot access the file because it is being used by another process
scons: building terminated because of errors.

We are running Python 2.6.5 (with pywin32) and SCons 2.3.6. This file is being copied due to a Install() call. Note that, for various historical reasons, we have the following decider for these Installs:

def _copy_decider(dependency, target, prev_ni):
    target = str(target.abspath)
    dependency = str(dependency.abspath)
    if os.path.isfile(target) and os.path.isfile(dependency):
        # By default, filecmp.cmp assumes that files with identical os.stat signatures
        # (which includes the inode) are the same file and, hence, must be the same.
        # However, on Windows, there is no inode - it appears to be set to zero - so
        # any two files with the same size and access/creation/modification times
        # will have the same os.stat signature, leading to a false positive. For this
        # reason, we must force it to do an actual file comparison by setting shallow
        # to False
        return not filecmp.cmp(target, dependency, shallow = False)
    else:
        # Either one of the dependency or target isn't a file or one of the files
        # (presumably the target) isn't there so do the copy
        return True

Note also that our IT department claims that virus checkers are disabled within the directory where the build is being performed (and we certainly have not seen any indication in the virus checker console to suggest that to be incorrect).

Does anyone have any thoughts as to what the problem might be?

Thanks in advance,

S.

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users



_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users


_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users




_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users



_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160719/ef8613f6/attachment-0001.html>


More information about the Scons-users mailing list