[Scons-users] Reading Python 3.8.0 .dblite file from Python 2.7

[Don Baldwin]

Sorry, all these version numbers are getting me a bit addled 😉.  We’re using SCons 3.1.1 in our new builds.

I can’t share the tools themselves unfortunately, but I can share some mock-up code along with the dblite files that illustrate the problem.  The code is attached to this email, and I’ll respond separately with the other files so as not to spam everyone else on this list.

I ran the code on three .dblite files:


  *   Py_2:   Using SCons 3.1.1 and Python 2.7.15
  *   Py_3:   Using SCons 3.1.1 and Python 3.8.0
  *   Py_3_w_fix:  Using SCons 3.1.1 and Python 3.8.0, but with the change in lib/SCons/compat/__init__.py to set the pickle protocol to 2.

In all three cases, the attached script was run under Python 2.7.15.

Py_2 and py_e_w_fix have identical output except for some ID and timestamp values.  Py_3 output is as follows:

sconsign: ignoring invalid `dblite' file `.sconsign.py_3.dblite': unsupported pickle protocol: 5
Python 2 only supports pickle protocols 0-2.

-Don

From: Bill Deegan <bill at baddogconsulting.com>
Sent: Wednesday, December 4, 2019 3:05 PM
To: Don Baldwin <donb at qti.qualcomm.com>
Cc: SCons users mailing list <scons-users at scons.org>
Subject: [EXT] Re: [Scons-users] Reading Python 3.8.0 .dblite file from Python 2.7
--force-build
There's no SCons 3.8.0.. (yet..) ;)
So you have tools which run sconsign and do something with the output?
If so then it shouldn't matter which python your tools run under as long as they use the same python to run sconsign which is used to run scons.

If not and you're unpickling the sconsign. Any chance you can share those tools?

On Wed, Dec 4, 2019 at 10:22 AM Don Baldwin <donb at qti.qualcomm.com<mailto:donb at qti.qualcomm.com>> wrote:
Hi Bill,

Thanks for your response.  Just to clarify though, I’m not trying to build the same build with both Python 2 and 3.  I have external tools that call sconsign to read the .dblite file to do analyses on it, and those tools are executed under Python 2.  Therefore, I don’t think I’ll see any rebuild issues.  My only issue is that the tools we use to read the dblite file are written for Python 2, and can’t support the newer pickle protocols.

We haven’t seen any issues trying to read .dblite files from SCons 3.8.0 builds that were run under Python 2.7, so we must not be looking at any of the objects that changed.  We’re mostly just looking at dependencies.

-Don


From: Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>>
Sent: Tuesday, December 3, 2019 9:35 PM
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Cc: Don Baldwin <donb at qti.qualcomm.com<mailto:donb at qti.qualcomm.com>>
Subject: [EXT] Re: [Scons-users] Reading Python 3.8.0 .dblite file from Python 2.7

Don,

Even if you use the same pickle versions you will likely still see some rebuilds as python 2 vs python 3 pickle some  objects differently. Also the signatures of python function actions will be different.
And this is explicitly specified in CHANGES.txt and the release notes for 3.0.0
You can specify the SConsign file name and add .2 and .3 on the end so at least they won't conflict with each other.

-Bill

On Tue, Dec 3, 2019 at 4:48 PM Mats Wichmann <mats at wichmann.us<mailto:mats at wichmann.us>> wrote:
On 12/3/19 5:35 PM, Don Baldwin wrote:
> Hi,
>

>     ignoring invalid `dblite' file
>
>  From what I've read, this seems to be due to the incompatibility
> between the pickle library between Python 2 and 3.

evolving protocols, yes. actually it's not just py 2/3, 3.8 added a new
pickle protocol that earlier Py3's can't read.

> I see the following code in lib/SCons/compat/__init__.py:
>
>     # Was pickle.HIGHEST_PROTOCOL
>     # Changed to 2 so py3.5+'s pickle will be compatible with py2.7.
>     PICKLE_PROTOCOL = pickle.HIGHEST_PROTOCOL
>
>
> When I modify this to set PICKLE_PROTOCOL = 2, my tools are able to
> successfully read the .dblite files, but I'd prefer not to have to
> modify the native SCons code.
>
> Is there a way to set this value from my SConscript file, or some other
> preferred way to handle this situation?
No.  SCons picks the highest available pickle version which is arguably
a dubious choice for your situation (and mine, it turns out), but that's
what it does.  There's no official API-ish way to request a different
one.  that would be a feature request, I guess.

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20191205/d1042748/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_sconsign.py
Type: application/octet-stream
Size: 287 bytes
Desc: test_sconsign.py
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20191205/d1042748/attachment-0001.obj>