[Scons-users] CacheDir race during parallel Windows builds?

[William Blevins]

Inlined comment.

On Mon, Aug 8, 2016 at 1:53 PM, Thomas Berg <merlin66b at gmail.com> wrote:

> Hi,
>
> On Mon, Aug 8, 2016 at 2:30 PM, Andrew C. Morrow
> <andrew.c.morrow at gmail.com> wrote:
> >
> >
> > On Sun, Aug 7, 2016 at 6:35 PM, Jason Kenny <dragon512 at live.com> wrote:
> >> Based on this I would suggest we have a race in SCons with cacheDir set
> in
> >> which python has a write mode handle open on the object file that was
> not
> >> closed yet. I did this test on Windows 10 with VS 2015 ( I tested linux
> on
> >> the bash shell feature on windows 10 and doubled checked on Ubuntu in a
> VM).
> >> The race I would assume to be something with the actions running a link
> >> command while the main thread is doing something with that file. Or
> there is
> >> something else touching that file.
> >>
> >>
> >>
> >> I don’t know enough of the pathways with cacheDir at the moment to say
> >> want would be going on.
> >
> >
> > Nor do I. I'm going to enlist the help of one of our local Windows
> experts
> > to see if he can help with tooling that will show us exactly what the
> > conflict is. I'll report back any findings.
> >
>
> Please ignore this if you don't think it's related, but it may be
> worth mentioning in case it is: in my work project we have had some
> apparent race conditions too, and we too have been suspecting a
> problem either in SCons or in Python's file handling.
>
> One of the problems was as follows: a builder written in python
> generates a C++ source file, using regular python file operations
> (tried both the 'with' statement and classic open / close). We would
> randomly get file access errors when the compiler tried to read the
> file immediately after generation. So it would seem that even though
> we closed the file in scons/python, it wasn't really properly closed.
>
> Another problem we have is that the compiler sometimes has problems
> accessing the C++ code generated by the Qt Moc process. In this case
> the file is created by an external process (run by scons), but maybe
> the file is also read and hashed by scons after it is created, so it
> could still be scons/python that is causing the access error? This I'm
> not sure about.
>

SCons will scan any generated files for implicit dependencies once they
have been created. This should be a read-only operation though. Perhaps, we
should double check, but Jason's observation about fopen() in Windows may
be the key here sadly.


>
> The difference from your case is that this is unrelated to cacheDir.
> We are not using this. The similarities are:
> - it's a file access problem where the file should be readable, but isn't
> - this happens only on Windows and only in parallel builds
> - it happens on virtual CI servers most of the time, I've rarely seen
> the problems on regular developer machines, similar to what was
> mentioned in this thread.
>
> We have not been able to debug it further yet. So we have not ruled
> out things like antivirus for example.
>
> Cheers,
> Thomas
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160808/470f31ef/attachment.html>