[Scons-users] scons-4.3.0 bug reading scons-3.1.2 generated .sconsign.dblite

Jay West jhdub23 at gmail.com
Mon Mar 21 14:56:26 EDT 2022 

I'm not asking for the sconsign file to be made compatible between
versions.  What I'm asking for is to ensure that scons refuses to create
bad builds.  A full rebuild is acceptable.  Error'ing out is acceptable.
Silently creating a bad build is not acceptable.

A developer may check out old code that must be run with scons2
(scons-3.1.2 with python2).  The developer may also check out the most
recent code that must run with scons3 (scons-4.3.0 with python3).  The
following happens:


   - Compile new code with scons3, followed by compiling old code with
   scons2.  scons2 errors out with a pickling error.  This is perfectly fine.
   The developer removes the .sconsign.dblite file and all is good.
   - Compile old code with scons2, followed by compiling new code with
   scon3.  The build is successful, but the build is corrupt.  *This is not
   acceptable.*


We have unknowingly deployed bad builds to the field, and this is causing
all sorts of havoc.  I have added the below to our SConstruct file for one
project.  Can something like this be added to scons?

Thanks,

Jay

    def fix_scons_db():
        '''
        Fix scons2 vs scons3 DB backward compatibility problem

        The dependency database from scons-3.1.2 cannot be read by
scons-4.3.0.  Instead of generating an error, scons-4.3.0 will blissfully
ignore the error and produce an incorrect build.
        In order to workaround this, we'll use use an scons
version-specific DB file.  Credit for the idea to:

https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022
        This solution also future proofs against further scons upgrades.
        '''
        scons_db_file_name = '.sconsign.{}'.format(SCons.__version__)
        SConsignFile(scons_db_file_name)
        scons_db_file_name += '.dblite' # scons adds this suffix
        scons_old_db_files = glob.glob('.sconsign.*')
        if scons_db_file_name in scons_old_db_files:
            scons_old_db_files.remove(scons_db_file_name)
        #print('scons DB file name:', scons_db_file_name, 'removing',
scons_old_db_files)
        if len(scons_old_db_files):
            subprocess.run(['rm -f .sconsign.*'], check=True, shell=True)


On Sun, Mar 20, 2022 at 9:36 PM Bill Deegan <bill at baddogconsulting.com>
wrote:

> There's no (reasonable) way to make the sconsign files be compatible
> between py3.5 and p2.7.
> The only way to not have invalid (as in crashes scons per your original
> email) is to name the sconsign files per python version.
> (Which I mentioned in my previous email, and is what the commit you
> pointed to).
>
> Of course as you discovered (most likely) you will rebuild items which
> were not out of date.
>
> The recommended way is to just validate the new scons & python combination
> on your build and then make the switch.
>
> -Bill
>
>
> On Sun, Mar 20, 2022 at 7:23 PM Jay West <jhdub23 at gmail.com> wrote:
>
>> Actually, the workaround isn't sufficient.  We also need to delete all
>> the other versions of .sconsign.dblite* as they will be out of date.
>>
>> Jay
>>
>>
>> On Sun, Mar 20, 2022 at 6:58 PM Jay West <jhdub23 at gmail.com> wrote:
>>
>>> If scons did a full rebuild, it would not be a problem.  The issue is
>>> that scons does NOT do a full rebuild, thinking that targets were up to
>>> date when they were really not.
>>>
>>> I see a workaround that someone else took:
>>> https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022.
>>> I will be putting this into our SContruct file.  Can this be made the
>>> default behavior?
>>>
>>> Thanks,
>>>
>>> Jay
>>>
>>>
>>> On Sat, Mar 19, 2022 at 11:09 AM Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>>> Switching from py2.7 -> py 3.5 invalidates the .sconsign files contents.
>>>> You have to do a full build when you make that switch.
>>>> It's in the release notes somewhere.
>>>>
>>>> https://scons.org/scons-300-is-available.html
>>>>
>>>> There's no way around it.
>>>> The best you can do is name your SConsign file to include the python
>>>> version so at least they won't be invalid when you switch from one to
>>>> another, but you will see additional rebuilds which may not be strictly
>>>> necessary.
>>>>
>>>> -Bill
>>>>
>>>> On Fri, Mar 18, 2022 at 5:03 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>
>>>>> Yes, our scons-3.1.2 was using python-2.7.17, and our scons-4.3.1 is
>>>>> using python-3.9.10.  We get a pickling error when trying to run
>>>>> scons-3.1.2 after scons-4.3.1 which is good.  Running scons-4.3.1 after
>>>>> scons-3.1.2 runs without error, but results in bad builds, which is
>>>>> terrible.
>>>>>
>>>>>
>>>>> On Fri, Mar 18, 2022 at 3:04 PM Bill Deegan <bill at baddogconsulting.com>
>>>>> wrote:
>>>>>
>>>>>> Sounds like you also switched from python 2.7 to python 3.5 or higher?
>>>>>>
>>>>>> On Fri, Mar 18, 2022 at 2:07 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>>>
>>>>>>> We are migrating from scons-3.1.2 to scons-4.3.0 and have
>>>>>>> encountered incorrect builds.
>>>>>>>
>>>>>>> After building with scons-3.1.2, changing some files, and then
>>>>>>> running scons-4.3.0, it appears that scons-4.3.0 cannot read some
>>>>>>> dependencies from the DB file.  Instead of error'ing out or ignoring the DB
>>>>>>> file, it happily generates an incorrect build.  However, running:
>>>>>>>
>>>>>>> sconsign .sconsign.dblite
>>>>>>>
>>>>>>>
>>>>>>> gives
>>>>>>>
>>>>>>> TypeError: '<' not supported between instances of 'str' and 'bytes'
>>>>>>>
>>>>>>>
>>>>>>> Has anyone seen this issue?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Jay
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20220321/d80c6fc6/attachment.htm>

More information about the Scons-users mailing list