[Scons-users] scons-4.3.0 bug reading scons-3.1.2 generated .sconsign.dblite
Bill Deegan
bill at baddogconsulting.com
Wed Mar 23 00:27:18 EDT 2022
Jay,
You're deploying non-full builds (incremental) to the field? (built in the
same sandbox as older versions?)
I would not consider that a good practice.
If we change the default sconsign naming we need to go through a
deprecation cycle (we need to notify on next release, and then roll a major
release to push the change), it will break lots of tests, and also affect
users expecting the sconsign file naming. And even if we did this if you
used a python 3.5+ with scons 3.0+ and did the same switch back and forth
that you're doing, the issue would not be resolved.
That said you are free to enable switching back and forth in your code by
naming the sconsign file to include the python version.
This would solve your issue.
Could you make a small reproducer? Perhaps there's another way to avoid the
bad outcome.
-Bill
On Mon, Mar 21, 2022 at 11:56 AM Jay West <jhdub23 at gmail.com> wrote:
> I'm not asking for the sconsign file to be made compatible between
> versions. What I'm asking for is to ensure that scons refuses to create
> bad builds. A full rebuild is acceptable. Error'ing out is acceptable.
> Silently creating a bad build is not acceptable.
>
> A developer may check out old code that must be run with scons2
> (scons-3.1.2 with python2). The developer may also check out the most
> recent code that must run with scons3 (scons-4.3.0 with python3). The
> following happens:
>
>
> - Compile new code with scons3, followed by compiling old code with
> scons2. scons2 errors out with a pickling error. This is perfectly fine.
> The developer removes the .sconsign.dblite file and all is good.
> - Compile old code with scons2, followed by compiling new code with
> scon3. The build is successful, but the build is corrupt. *This is
> not acceptable.*
>
>
> We have unknowingly deployed bad builds to the field, and this is causing
> all sorts of havoc. I have added the below to our SConstruct file for one
> project. Can something like this be added to scons?
>
> Thanks,
>
> Jay
>
> def fix_scons_db():
> '''
> Fix scons2 vs scons3 DB backward compatibility problem
>
> The dependency database from scons-3.1.2 cannot be read by
> scons-4.3.0. Instead of generating an error, scons-4.3.0 will blissfully
> ignore the error and produce an incorrect build.
> In order to workaround this, we'll use use an scons
> version-specific DB file. Credit for the idea to:
>
> https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022
> This solution also future proofs against further scons upgrades.
> '''
> scons_db_file_name = '.sconsign.{}'.format(SCons.__version__)
> SConsignFile(scons_db_file_name)
> scons_db_file_name += '.dblite' # scons adds this suffix
> scons_old_db_files = glob.glob('.sconsign.*')
> if scons_db_file_name in scons_old_db_files:
> scons_old_db_files.remove(scons_db_file_name)
> #print('scons DB file name:', scons_db_file_name, 'removing',
> scons_old_db_files)
> if len(scons_old_db_files):
> subprocess.run(['rm -f .sconsign.*'], check=True, shell=True)
>
>
> On Sun, Mar 20, 2022 at 9:36 PM Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> There's no (reasonable) way to make the sconsign files be compatible
>> between py3.5 and p2.7.
>> The only way to not have invalid (as in crashes scons per your original
>> email) is to name the sconsign files per python version.
>> (Which I mentioned in my previous email, and is what the commit you
>> pointed to).
>>
>> Of course as you discovered (most likely) you will rebuild items which
>> were not out of date.
>>
>> The recommended way is to just validate the new scons & python
>> combination on your build and then make the switch.
>>
>> -Bill
>>
>>
>> On Sun, Mar 20, 2022 at 7:23 PM Jay West <jhdub23 at gmail.com> wrote:
>>
>>> Actually, the workaround isn't sufficient. We also need to delete all
>>> the other versions of .sconsign.dblite* as they will be out of date.
>>>
>>> Jay
>>>
>>>
>>> On Sun, Mar 20, 2022 at 6:58 PM Jay West <jhdub23 at gmail.com> wrote:
>>>
>>>> If scons did a full rebuild, it would not be a problem. The issue is
>>>> that scons does NOT do a full rebuild, thinking that targets were up to
>>>> date when they were really not.
>>>>
>>>> I see a workaround that someone else took:
>>>> https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022.
>>>> I will be putting this into our SContruct file. Can this be made the
>>>> default behavior?
>>>>
>>>> Thanks,
>>>>
>>>> Jay
>>>>
>>>>
>>>> On Sat, Mar 19, 2022 at 11:09 AM Bill Deegan <bill at baddogconsulting.com>
>>>> wrote:
>>>>
>>>>> Switching from py2.7 -> py 3.5 invalidates the .sconsign files
>>>>> contents.
>>>>> You have to do a full build when you make that switch.
>>>>> It's in the release notes somewhere.
>>>>>
>>>>> https://scons.org/scons-300-is-available.html
>>>>>
>>>>> There's no way around it.
>>>>> The best you can do is name your SConsign file to include the python
>>>>> version so at least they won't be invalid when you switch from one to
>>>>> another, but you will see additional rebuilds which may not be strictly
>>>>> necessary.
>>>>>
>>>>> -Bill
>>>>>
>>>>> On Fri, Mar 18, 2022 at 5:03 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>>
>>>>>> Yes, our scons-3.1.2 was using python-2.7.17, and our scons-4.3.1 is
>>>>>> using python-3.9.10. We get a pickling error when trying to run
>>>>>> scons-3.1.2 after scons-4.3.1 which is good. Running scons-4.3.1 after
>>>>>> scons-3.1.2 runs without error, but results in bad builds, which is
>>>>>> terrible.
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 18, 2022 at 3:04 PM Bill Deegan <
>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>
>>>>>>> Sounds like you also switched from python 2.7 to python 3.5 or
>>>>>>> higher?
>>>>>>>
>>>>>>> On Fri, Mar 18, 2022 at 2:07 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>>>>
>>>>>>>> We are migrating from scons-3.1.2 to scons-4.3.0 and have
>>>>>>>> encountered incorrect builds.
>>>>>>>>
>>>>>>>> After building with scons-3.1.2, changing some files, and then
>>>>>>>> running scons-4.3.0, it appears that scons-4.3.0 cannot read some
>>>>>>>> dependencies from the DB file. Instead of error'ing out or ignoring the DB
>>>>>>>> file, it happily generates an incorrect build. However, running:
>>>>>>>>
>>>>>>>> sconsign .sconsign.dblite
>>>>>>>>
>>>>>>>>
>>>>>>>> gives
>>>>>>>>
>>>>>>>> TypeError: '<' not supported between instances of 'str' and 'bytes'
>>>>>>>>
>>>>>>>>
>>>>>>>> Has anyone seen this issue?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Jay
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20220322/fda7e7cf/attachment.htm>
More information about the Scons-users
mailing list