[Scons-users] CacheDir race during parallel Windows builds?

[Thomas Berg]

Hi,

On Mon, Aug 8, 2016 at 2:30 PM, Andrew C. Morrow
<andrew.c.morrow at gmail.com> wrote:
>
>
> On Sun, Aug 7, 2016 at 6:35 PM, Jason Kenny <dragon512 at live.com> wrote:
>> Based on this I would suggest we have a race in SCons with cacheDir set in
>> which python has a write mode handle open on the object file that was not
>> closed yet. I did this test on Windows 10 with VS 2015 ( I tested linux on
>> the bash shell feature on windows 10 and doubled checked on Ubuntu in a VM).
>> The race I would assume to be something with the actions running a link
>> command while the main thread is doing something with that file. Or there is
>> something else touching that file.
>>
>>
>>
>> I don’t know enough of the pathways with cacheDir at the moment to say
>> want would be going on.
>
>
> Nor do I. I'm going to enlist the help of one of our local Windows experts
> to see if he can help with tooling that will show us exactly what the
> conflict is. I'll report back any findings.
>

Please ignore this if you don't think it's related, but it may be
worth mentioning in case it is: in my work project we have had some
apparent race conditions too, and we too have been suspecting a
problem either in SCons or in Python's file handling.

One of the problems was as follows: a builder written in python
generates a C++ source file, using regular python file operations
(tried both the 'with' statement and classic open / close). We would
randomly get file access errors when the compiler tried to read the
file immediately after generation. So it would seem that even though
we closed the file in scons/python, it wasn't really properly closed.

Another problem we have is that the compiler sometimes has problems
accessing the C++ code generated by the Qt Moc process. In this case
the file is created by an external process (run by scons), but maybe
the file is also read and hashed by scons after it is created, so it
could still be scons/python that is causing the access error? This I'm
not sure about.

The difference from your case is that this is unrelated to cacheDir.
We are not using this. The similarities are:
- it's a file access problem where the file should be readable, but isn't
- this happens only on Windows and only in parallel builds
- it happens on virtual CI servers most of the time, I've rarely seen
the problems on regular developer machines, similar to what was
mentioned in this thread.

We have not been able to debug it further yet. So we have not ruled
out things like antivirus for example.

Cheers,
Thomas