[Scons-users] Out of memory writing back the database

[Bill Deegan]

Steve,

How many files are you processing? 500MB is very large for a sconsign.

Could you try mv .sconsign .sconsign.save, let your build run and see how
big the new file is?

-Bill

On Tue, Nov 28, 2017 at 7:17 AM, Hill, Steve (FP COM) <Steve.Hill at cobham.com
> wrote:

> All,
>
> We are using 32-bit Python 2.7.12 and SCons 2.5.1 on Windows (and Linux).
> We have now reached the case where the database is in excess of 500MB.
> During the build, the memory usage bubbles along at just over 1GB but, at
> the end of the run we sometimes get a spike in memory usage that causes an
> "Out of memory" exception and the build fails.
>
> Looking at the code, it appears that the database file is produced by
> pickling into memory before writing to the file, hence the memory usage
> increases by over 0.5GB as it is being written back  - and, depending on
> the database size, this can blow the memory space of a 32-bit process.
>
> Firstly, I do have a medium term plan to move us to 64-bit Python with the
> move to Python 3 but we still need to stick with 32-bit Python at the
> moment (for example, we have to load 32-bit DLLs) so I need to fix this for
> 32-bit Python in the short term.
>
> Secondly, I've had a look at the old (?) .sconsign per directory approach,
> which would seem to address this problem. I assume that this is still
> supported and should be as robust as the single file approach?
>
> The only issue with this is that some of the .sconsigns appear in the
> directories beside the source code, which breaches the contract for our
> build system, where no file may be produced within the directories
> containing source code. Where we use VariantDirs (for the actual
> implementation files), there is no problem but, for header files, the
> .sconsign ends up in the include or interface directory. I can see how I
> can reasonably easily monkey patch the DirFile class in SConsign.py to
> override the directory in which the .sconsign resides but, before I do
> this, I wanted to check that there is no easier way to achieve the builds
> without a memory issue.
>
> Thanks,
>
> Steve.
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20171130/a80d5e24/attachment-0001.html>