[Scons-users] Godot generated dependency files access issue

Thu Jun 16 05:56:51 EDT 2016

Aleksandar,

I assume you mean SCons 2.5.0 (since there isn't a 2.5.1). The primary
change to SCons 2.5.0 was correcting cross-language implicit scanning not
honoring scanner keys. This means there may be a different in your
dependency tree between 2.4.1 and 2.5.0 (with 2.5.0 correctly having new
dependencies).

See more inline comments.

On Thu, Jun 16, 2016 at 9:32 AM, Alexandar Danilovic <
greatgames.alexandar at gmail.com> wrote:

> Hello, folks.
>
> Godot game engine has a Scons build issue and I am hoping kind folks here
> can offer some help.
>
> Godot has for a long time had an issue with it's parallel builds on
> windows with MSVC...
>
> Basically, generated .h files are immediately after creation required by
> the next compile order and cl.exe cannot access it immediately after it is
> generated...
>
> ...\drivers/gles2/rasterizer_gles2.h(55): fatal error C1083: Cannot open
> include file: 'drivers/gles2/shaders/canvas.glsl.h': No such file or
> directory
> 1>  scons: *** [platform\windows\os_windows.windows.tools.64.obj] Error 2
>
> Now, i have tested whether they are in the dependency tree, and they
> are... Which is why -j1 builds work fine... only -j2+ fail...
>
> Another important bit of info is that parallel builds work fine with Scons
> 2.4.1 and 2.4.0 when Pywin32 is installed... when pywin isn't installed,
> they fail with this error just like 2.5.1...
>
> But 2.5.1, the latest one... does not compile parallel builds at all...
> even though i have Pywin32 installed for it (it's using the same python
> install)... it errors on the above message regardless of pywin32.
>

As you state below, if its related to file handles not closing, then this
is simply a race condition. I know that 2.5.0 is a little slower on the
dependency scan, so this could make the difference. There is an
optimization for the cross-language changes currently in pull request form,
but that isn't the root issue.
https://bitbucket.org/scons/scons/pull-requests/345/optimize-implicit-dependency-scan/diff

>
> Now, i managed to get compiles to compile on 2.5.1 with a hack...
> basically, i sleep the file builders that build the generated .h files for
> a few seconds to allow file handles to be closed in time (i believe it's
> the handles)...
>
>
> https://github.com/Griefchief/godot/commit/4c3b9c2a71b69ec5922bdd47a5bde4acfe09a477#diff-45b856a77c4dd4017d6ebe6299a75cb4R1463
>
>
> I want to ask if any of you kind folks that have read this far have any
> idea if there is a "propper", non hackish fix for this?
>
> I did this fix because i needed to be sure what was going on, so that you
> and I understand better what is happening...
>
> I will note again what i have noticed:
>
> 1) Generated .h files have access issues, my guess is due to non released
> file handles
> 2) File handles should be released by python either manually via close()
> or during garbage calls...
> 3) File handle release is not happening fast enough... for a really huge
> file called certs.h that has several MBs i noticed that it seems to take
> longer then for the small files used for shaders etc that are kilobytes in
> size... i had to increase the wait for this file to 4 seconds instead of 1
> or 2 to ensure it will not error out... Writes should not be asynchronous
> so i'm not sure what is happening... maybe it was a fluke that certs.h took
> longer to release the handle...
>
> 4) Pywin32 fixes 2.4.1 and 2.4.0 versions of SCons... however, for 2.5.1
> Pywin32 installed seems to have no effect. I am not the only person that
> has issues only with 2.5.1 (when pywin is installed). Other folks have also
> reported only 2.5.1 being the problem.
>

I know that SCons generally expects the garbage collector to handle closing
files. This has caused some problems in general, and there are some pending
pull requests which will explicitly close them after access which should
fix some of the issues here.
https://bitbucket.org/scons/scons/pull-requests/347/avoid-using-__slots__-on-node-and-executor/diff

>
> The above points more or less cover it. Even though i attempt to close the
> handles manually, they simply seem to not be released in time...
>
> It seems that there is some kind of interaction between the
> filesystem/processes or something third that keeps the file handles alive
> for too long. It's strange that it is actually sleep() that releases the
> handles in time... if Python was leaking handles, I'd expect sleep() to not
> have an effect but it does... I have zero experience with file handles,
> maybe something else is going on that needs time that i don't know about
> that is not Python itself (windows related?)...
>
> I hope i have given you enough information to offer your suggestions if
> you have any.
>
> Again, 2.5.1 seems to be a hotspot...
>
> Best regards,
> Aleksandar Danilovic.
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160616/5b91ce24/attachment.html>