[Scons-users] Any compatibility breaking changes to SConsign DB / MemoryError in 4.1.0?

[Bill Deegan]

Markus,

There will never be any guarantee of backwards compatibility for sconsign
files.
Generally there will be a notice in the RELEASE/CHANGES if we break
compatibility going forward.

The note you're referring to was due to defaulting to the newest pickle
version which got bumped up in 3.8, we dropped it back down to one which is
common for all supported pythons.

That said, given your stack trace I'd say it's very unlikely that it has
anything to do with sconsign.
A guess based on your stack trace makes me thing you have a VERY large file
which scons is trying to read into memory and running out of memory.
Have you changed File.md5_chunksize?
If you run top while running this build, can you see if you're running out
of memory?
Is your build perhaps linking some large binaries (perhaps in parallel)
when you get this error?

You've not stated which version of Python you're using?
Can you try using SCons 4.2.0?

Can you create a small reproducer?

-Bill
SCons Project Co-Manager

On Mon, Sep 27, 2021 at 8:56 AM Markus B. Schenkel <mbschenkel at gmail.com>
wrote:

> Hello,
>
> We recently upgraded from SCons 3.1.2 to 4.1.0 and are facing some
> "MemoryErrors" at the end of our builds (for completeness, the stack-trace
> is in the end).
> One hypothesis would attribute this to .sconsign.dblite incompatibilities.
> Before we now introduce some clever version handling logic to purge those
> files, I would like to know whether there were any changes breaking both
> forward (as well as backward compatibility - in case we have to revert) of
> the sconsign db files and if they would be listed in the release notes?
>
> The only related entry in the release notes that I could find is for 4.0.0:
> "NOTE: If you used Python 3.8 with SCons 3.0.0 or above, you may get a a
> pickle protocol error. Remove your .sconsign.dblite. You will end up with a
> full rebuild."
> But we did not get such pickle errors.
>
> Thanks,
> Markus
>
>
> --------------------------------------------------------------------------------
> -- stack-trace of MemoryError:
>
> --------------------------------------------------------------------------------
>   scons: done building targets.
> MemoryError: :
>   File "<build_path>\venv\lib\site-packages\SCons\Script\Main.py", line
> 1380:
>     _exec_main(parser, values)
>   File "<build_path>\venv\lib\site-packages\SCons\Script\Main.py", line
> 1344:
>     _main(parser)
>   File "<build_path>\venv\lib\site-packages\SCons\Script\Main.py", line
> 1113:
>     nodes = _build_targets(fs, options, targets, target_top)
>   File "<build_path>\venv\lib\site-packages\SCons\Script\Main.py", line
> 1319:
>     jobs.run(postfunc = jobs_postfunc)
>   File "<build_path>\venv\lib\site-packages\SCons\Job.py", line 107:
>     self.job.start()
>   File "<build_path>\venv\lib\site-packages\SCons\Job.py", line 406:
>     task.executed()
>   File "<build_path>\venv\lib\site-packages\SCons\Script\Main.py", line
> 242:
>     SCons.Taskmaster.OutOfDateTask.executed(self)
>   File "<build_path>\venv\lib\site-packages\SCons\Taskmaster.py", line 293:
>     t.built()
>   File "<build_path>\venv\lib\site-packages\SCons\Node\FS.py", line 3250:
>     SCons.Node.Node.built(self)
>   File "<build_path>\venv\lib\site-packages\SCons\Node\__init__.py", line
> 797:
>     self.ninfo.update(self)
>   File "<build_path>\venv\lib\site-packages\SCons\Node\__init__.py", line
> 372:
>     setattr(self, f, func())
>   File "<build_path>\venv\lib\site-packages\SCons\Node\FS.py", line 3215:
>     contents = self.get_contents()
>   File "<build_path>\venv\lib\site-packages\SCons\Node\FS.py", line 2698:
>     return SCons.Node._get_contents_map[self._func_get_contents](self)
>   File "<build_path>\venv\lib\site-packages\SCons\Node\__init__.py", line
> 215:
>     contents = fp.read()
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210927/1b5a8556/attachment.htm>