[Scons-users] Anyone depending on having per directory SConsign files

[Mats Wichmann]

On 11/26/20 12:53 AM, Steve Hill (Wireless) via Scons-users wrote:
> Hi Bill,
> 
> Yes, we do use this – but the use-case is a little complex…
> 
> It started when we were still running with a 32-bit version of Python. 
> Since the code to write back the SConsign assembles it in memory first, 
> and our SConsign was many hundreds of MB in size, it was sometimes 
> causing an out-of-memory exception.
> 
> Hence, what I did was to create a patched version of the DirFile class 
> that wrote back a file per directory but named with a hash of the 
> directory name and stored in a single sub-directory below the 
> SConstruct, and patched SCons with that class. I then used 
> SConsignFile(None) to invoke the new behaviour.
> 
> This solved the out-of-memory issue and, even when we moved to a 64-bit 
> version of Python, it was a performance improvement since, if a dev was 
> only building a small proportion of the overall codebase (a common 
> use-case – just running the unit-test suite for a single source file 
> when TDDing, for instance), writing back a few relatively small files 
> was often much quicker than writing back a several hundred MB file.

That's not so wild.  There's been an intent for a while (not backed up 
by any pull request or experimental branch so far) to switch to a more 
database-like model for the signature stuff, where records can be 
read/written individually, rather than a huge dump of everything that's 
previously all been kept in memory.   Hundreds of MB of data to pickle 
in one go is a pretty scary thought even with the improvements that have 
happened to pickle performance in Python....