[Scons-users] Godot generated dependency files access issue

[Bill Deegan]

Aleksandar,

I chatted with someone on IRC about exactly this issue.
I believe the dependency is not captured properly and so the header files
are not generated prior to the compile (which is failing) is started.
If you run scons --tree=prune and capture the output you'll see that the
source file for canvas.glsl.h is not listed as a dependency.  (If I
remember correctly that's a generated file)

-Bill



On Thu, Jun 16, 2016 at 5:56 AM, William Blevins <wblevins001 at gmail.com>
wrote:

> Aleksandar,
>
> I assume you mean SCons 2.5.0 (since there isn't a 2.5.1). The primary
> change to SCons 2.5.0 was correcting cross-language implicit scanning not
> honoring scanner keys. This means there may be a different in your
> dependency tree between 2.4.1 and 2.5.0 (with 2.5.0 correctly having new
> dependencies).
>
> See more inline comments.
>
> On Thu, Jun 16, 2016 at 9:32 AM, Alexandar Danilovic <
> greatgames.alexandar at gmail.com> wrote:
>
>> Hello, folks.
>>
>> Godot game engine has a Scons build issue and I am hoping kind folks here
>> can offer some help.
>>
>> Godot has for a long time had an issue with it's parallel builds on
>> windows with MSVC...
>>
>> Basically, generated .h files are immediately after creation required by
>> the next compile order and cl.exe cannot access it immediately after it is
>> generated...
>>
>> ...\drivers/gles2/rasterizer_gles2.h(55): fatal error C1083: Cannot open
>> include file: 'drivers/gles2/shaders/canvas.glsl.h': No such file or
>> directory
>> 1>  scons: *** [platform\windows\os_windows.windows.tools.64.obj] Error 2
>>
>> Now, i have tested whether they are in the dependency tree, and they
>> are... Which is why -j1 builds work fine... only -j2+ fail...
>>
>> Another important bit of info is that parallel builds work fine with
>> Scons 2.4.1 and 2.4.0 when Pywin32 is installed... when pywin isn't
>> installed, they fail with this error just like 2.5.1...
>>
>> But 2.5.1, the latest one... does not compile parallel builds at all...
>> even though i have Pywin32 installed for it (it's using the same python
>> install)... it errors on the above message regardless of pywin32.
>>
>
> As you state below, if its related to file handles not closing, then this
> is simply a race condition. I know that 2.5.0 is a little slower on the
> dependency scan, so this could make the difference. There is an
> optimization for the cross-language changes currently in pull request form,
> but that isn't the root issue.
> https://bitbucket.org/scons/scons/pull-requests/345/optimize-implicit-dependency-scan/diff
>
>
>>
>> Now, i managed to get compiles to compile on 2.5.1 with a hack...
>> basically, i sleep the file builders that build the generated .h files for
>> a few seconds to allow file handles to be closed in time (i believe it's
>> the handles)...
>>
>>
>> https://github.com/Griefchief/godot/commit/4c3b9c2a71b69ec5922bdd47a5bde4acfe09a477#diff-45b856a77c4dd4017d6ebe6299a75cb4R1463
>>
>>
>> I want to ask if any of you kind folks that have read this far have any
>> idea if there is a "propper", non hackish fix for this?
>>
>> I did this fix because i needed to be sure what was going on, so that you
>> and I understand better what is happening...
>>
>> I will note again what i have noticed:
>>
>> 1) Generated .h files have access issues, my guess is due to non released
>> file handles
>> 2) File handles should be released by python either manually via close()
>> or during garbage calls...
>> 3) File handle release is not happening fast enough... for a really huge
>> file called certs.h that has several MBs i noticed that it seems to take
>> longer then for the small files used for shaders etc that are kilobytes in
>> size... i had to increase the wait for this file to 4 seconds instead of 1
>> or 2 to ensure it will not error out... Writes should not be asynchronous
>> so i'm not sure what is happening... maybe it was a fluke that certs.h took
>> longer to release the handle...
>>
>> 4) Pywin32 fixes 2.4.1 and 2.4.0 versions of SCons... however, for 2.5.1
>> Pywin32 installed seems to have no effect. I am not the only person that
>> has issues only with 2.5.1 (when pywin is installed). Other folks have also
>> reported only 2.5.1 being the problem.
>>
>
> I know that SCons generally expects the garbage collector to handle
> closing files. This has caused some problems in general, and there are some
> pending pull requests which will explicitly close them after access which
> should fix some of the issues here.
> https://bitbucket.org/scons/scons/pull-requests/347/avoid-using-__slots__-on-node-and-executor/diff
>
>
>>
>> The above points more or less cover it. Even though i attempt to close
>> the handles manually, they simply seem to not be released in time...
>>
>> It seems that there is some kind of interaction between the
>> filesystem/processes or something third that keeps the file handles alive
>> for too long. It's strange that it is actually sleep() that releases the
>> handles in time... if Python was leaking handles, I'd expect sleep() to not
>> have an effect but it does... I have zero experience with file handles,
>> maybe something else is going on that needs time that i don't know about
>> that is not Python itself (windows related?)...
>>
>> I hope i have given you enough information to offer your suggestions if
>> you have any.
>>
>> Again, 2.5.1 seems to be a hotspot...
>>
>> Best regards,
>> Aleksandar Danilovic.
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160616/7c324a42/attachment-0001.html>