[Scons-users] Out of memory writing back the database

[Mats Wichmann]

Transition: json won't help with that. Sqlite might.

On December 1, 2017 9:19:44 AM MST, Bill Deegan <bill at baddogconsulting.com> wrote:
>Would you consider contributing that logic in a pull request and/or
>pointing to a commit(s) in your repo?
>
>Currently in the plans is to transition the .sconsign to another
>format,
>possibly json or sqlite with an eye on speed and/or incrementality.
>
>-Bill
>
>On Fri, Dec 1, 2017 at 5:21 AM, Hill, Steve (FP COM)
><Steve.Hill at cobham.com>
>wrote:
>
>> Hi Bill,
>>
>>
>>
>> Just to follow up on this. I’ve just monkey patched the DirFile class
>in
>> SConsign.py so that it writes the sconsign into a single temporary
>> directory with each sconsign named with an adler32 hash of the
>directory
>> path.
>>
>>
>>
>> After doing the 4 basic builds, the directory contains 10,745
>sconsigns
>> totalling 674MB. Using –debug=memory, the peak memory usage of the
>worst
>> case build was 925 MB, compared to ~1.4GB for the same build with the
>> monolithic sconsign (so the difference is basically the cost of
>pickling
>> the sconsign into memory first).
>>
>>
>>
>> Cheers,
>>
>>
>>
>> S.
>>
>>
>>
>>
>>
>> Hi Bill,
>>
>>
>>
>> Our code base is not that big – about 20,000 files, but most files
>get
>> built multiple times (for different CPUs) and we build it in several
>> different ways, each with many of their own VariantDirs. When the 4
>most
>> common builds are done, the .sconsign (having been deleted first) is
>just
>> under 500MB but, given a bit of history or add in some of the less
>common
>> builds, and it gets big enough to blow the memory space of Python.
>>
>>
>>
>> I’ve split the .sconsign up so that largely independent builds on the
>same
>> codebase are stored in a different .sconsign (initially done due to
>the
>> time taken to write the .sconsign back), though I’ve had to write
>some
>> custom Deciders to make this totally reliable. All in all, the
>.sconsign
>> per directory (though with a .sconsign location outside the main
>source
>> tree) would seem the best solution for me (and seems like it might be
>> better performing too as not all files will need writing back for
>> incremental builds), unless there is another option that I’m not
>aware of…
>>
>>
>>
>> Thanks,
>>
>>
>>
>> S.
>>
>>
>>
>>
>>
>>
>>
>> Steve,
>>
>>
>>
>> How many files are you processing? 500MB is very large for a
>sconsign.
>>
>>
>>
>> Could you try mv .sconsign .sconsign.save, let your build run and see
>how
>> big the new file is?
>>
>>
>>
>> -Bill
>>
>>
>>
>> On Tue, Nov 28, 2017 at 7:17 AM, Hill, Steve (FP COM) <
>> Steve.Hill at cobham.com> wrote:
>>
>> All,
>>
>> We are using 32-bit Python 2.7.12 and SCons 2.5.1 on Windows (and
>Linux).
>> We have now reached the case where the database is in excess of
>500MB.
>> During the build, the memory usage bubbles along at just over 1GB
>but, at
>> the end of the run we sometimes get a spike in memory usage that
>causes an
>> "Out of memory" exception and the build fails.
>>
>> Looking at the code, it appears that the database file is produced by
>> pickling into memory before writing to the file, hence the memory
>usage
>> increases by over 0.5GB as it is being written back  - and, depending
>on
>> the database size, this can blow the memory space of a 32-bit
>process.
>>
>> Firstly, I do have a medium term plan to move us to 64-bit Python
>with the
>> move to Python 3 but we still need to stick with 32-bit Python at the
>> moment (for example, we have to load 32-bit DLLs) so I need to fix
>this for
>> 32-bit Python in the short term.
>>
>> Secondly, I've had a look at the old (?) .sconsign per directory
>approach,
>> which would seem to address this problem. I assume that this is still
>> supported and should be as robust as the single file approach?
>>
>> The only issue with this is that some of the .sconsigns appear in the
>> directories beside the source code, which breaches the contract for
>our
>> build system, where no file may be produced within the directories
>> containing source code. Where we use VariantDirs (for the actual
>> implementation files), there is no problem but, for header files, the
>> .sconsign ends up in the include or interface directory. I can see
>how I
>> can reasonably easily monkey patch the DirFile class in SConsign.py
>to
>> override the directory in which the .sconsign resides but, before I
>do
>> this, I wanted to check that there is no easier way to achieve the
>builds
>> without a memory issue.
>>
>> Thanks,
>>
>> Steve.
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20171201/6fb89384/attachment-0001.html>