[Scons-users] Out of memory writing back the database

[Hill, Steve (FP COM)]

All,

We are using 32-bit Python 2.7.12 and SCons 2.5.1 on Windows (and Linux). We have now reached the case where the database is in excess of 500MB. During the build, the memory usage bubbles along at just over 1GB but, at the end of the run we sometimes get a spike in memory usage that causes an "Out of memory" exception and the build fails.

Looking at the code, it appears that the database file is produced by pickling into memory before writing to the file, hence the memory usage increases by over 0.5GB as it is being written back  - and, depending on the database size, this can blow the memory space of a 32-bit process.

Firstly, I do have a medium term plan to move us to 64-bit Python with the move to Python 3 but we still need to stick with 32-bit Python at the moment (for example, we have to load 32-bit DLLs) so I need to fix this for 32-bit Python in the short term.

Secondly, I've had a look at the old (?) .sconsign per directory approach, which would seem to address this problem. I assume that this is still supported and should be as robust as the single file approach?

The only issue with this is that some of the .sconsigns appear in the directories beside the source code, which breaches the contract for our build system, where no file may be produced within the directories containing source code. Where we use VariantDirs (for the actual implementation files), there is no problem but, for header files, the .sconsign ends up in the include or interface directory. I can see how I can reasonably easily monkey patch the DirFile class in SConsign.py to override the directory in which the .sconsign resides but, before I do this, I wanted to check that there is no easier way to achieve the builds without a memory issue.

Thanks,

Steve.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 526 bytes
Desc: not available
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20171128/fbe0f808/attachment.pgp>