[Scons-users] Out of memory writing back the database

[Hill, Steve (FP COM)]

Hi Bill,

 

Our code base is not that big – about 20,000 files, but most files get built multiple times (for different CPUs) and we build it in several different ways, each with many of their own VariantDirs. When the 4 most common builds are done, the .sconsign (having been deleted first) is just under 500MB but, given a bit of history or add in some of the less common builds, and it gets big enough to blow the memory space of Python.

 

I’ve split the .sconsign up so that largely independent builds on the same codebase are stored in a different .sconsign (initially done due to the time taken to write the .sconsign back), though I’ve had to write some custom Deciders to make this totally reliable. All in all, the .sconsign per directory (though with a .sconsign location outside the main source tree) would seem the best solution for me (and seems like it might be better performing too as not all files will need writing back for incremental builds), unless there is another option that I’m not aware of…

 

Thanks,

 

S.

 

 

 

Steve,

 

How many files are you processing? 500MB is very large for a sconsign.

 

Could you try mv .sconsign .sconsign.save, let your build run and see how big the new file is?

 

-Bill

 

On Tue, Nov 28, 2017 at 7:17 AM, Hill, Steve (FP COM) <Steve.Hill at cobham.com> wrote:

All,

We are using 32-bit Python 2.7.12 and SCons 2.5.1 on Windows (and Linux). We have now reached the case where the database is in excess of 500MB. During the build, the memory usage bubbles along at just over 1GB but, at the end of the run we sometimes get a spike in memory usage that causes an "Out of memory" exception and the build fails.

Looking at the code, it appears that the database file is produced by pickling into memory before writing to the file, hence the memory usage increases by over 0.5GB as it is being written back  - and, depending on the database size, this can blow the memory space of a 32-bit process.

Firstly, I do have a medium term plan to move us to 64-bit Python with the move to Python 3 but we still need to stick with 32-bit Python at the moment (for example, we have to load 32-bit DLLs) so I need to fix this for 32-bit Python in the short term.

Secondly, I've had a look at the old (?) .sconsign per directory approach, which would seem to address this problem. I assume that this is still supported and should be as robust as the single file approach?

The only issue with this is that some of the .sconsigns appear in the directories beside the source code, which breaches the contract for our build system, where no file may be produced within the directories containing source code. Where we use VariantDirs (for the actual implementation files), there is no problem but, for header files, the .sconsign ends up in the include or interface directory. I can see how I can reasonably easily monkey patch the DirFile class in SConsign.py to override the directory in which the .sconsign resides but, before I do this, I wanted to check that there is no easier way to achieve the builds without a memory issue.

Thanks,

Steve.


_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20171201/11f84301/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 526 bytes
Desc: not available
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20171201/11f84301/attachment.pgp>