[Scons-users] scons-4.3.0 bug reading scons-3.1.2 generated .sconsign.dblite
Bill Deegan
bill at baddogconsulting.com
Fri Apr 1 16:34:03 EDT 2022
A (small) reproducer would be helpful.
I've never seen a py27 sconsign silently do this.
Usually it will either rebuild everything, or error out.
So you just got "lucky" (not the good luck).
Regardless, we should sort this out.
I'm not sure how easy adding a version to the sconsign file would be.
I'll have to dig back into that logic.
-Bill
On Thu, Mar 24, 2022 at 12:22 AM Jay West <jhdub23 at gmail.com> wrote:
> "Deploy" was a poor choice of words. We are a large organization with a
> pipeline of builds and quality. Our fastest turnaround/lowest quality
> guarantees are quick development builds for preliminary testing of targeted
> features (non-production use, obviously). We are long-time users of scons
> and rely on the build correctness that scons can usually provide. I
> haven't seen another build tool that tries as hard as scons to encode even
> the commands and the build environment as dependencies. That's why I was
> surprised that a DB compatibility check wasn't in the code.
>
> I think an alternative solution of writing the scons version, or some
> other compatibility version information, as a header in the
> .sconsign.dblite would be much better than renaming files. If the header
> is not recognized as being compatible, nuke the file and start over. Is a
> reproducer really needed? Even conceptually, an incompatible DB format
> between versions is a known weakness.
>
> Thanks,
>
> Jay
>
>
> On Tue, Mar 22, 2022 at 10:44 PM Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> Jay,
>>
>> You're deploying non-full builds (incremental) to the field? (built in
>> the same sandbox as older versions?)
>> I would not consider that a good practice.
>>
>> If we change the default sconsign naming we need to go through a
>> deprecation cycle (we need to notify on next release, and then roll a major
>> release to push the change), it will break lots of tests, and also affect
>> users expecting the sconsign file naming. And even if we did this if you
>> used a python 3.5+ with scons 3.0+ and did the same switch back and forth
>> that you're doing, the issue would not be resolved.
>>
>> That said you are free to enable switching back and forth in your code by
>> naming the sconsign file to include the python version.
>> This would solve your issue.
>>
>> Could you make a small reproducer? Perhaps there's another way to avoid
>> the bad outcome.
>>
>> -Bill
>>
>>
>> On Mon, Mar 21, 2022 at 11:56 AM Jay West <jhdub23 at gmail.com> wrote:
>>
>>> I'm not asking for the sconsign file to be made compatible between
>>> versions. What I'm asking for is to ensure that scons refuses to create
>>> bad builds. A full rebuild is acceptable. Error'ing out is acceptable.
>>> Silently creating a bad build is not acceptable.
>>>
>>> A developer may check out old code that must be run with scons2
>>> (scons-3.1.2 with python2). The developer may also check out the most
>>> recent code that must run with scons3 (scons-4.3.0 with python3). The
>>> following happens:
>>>
>>>
>>> - Compile new code with scons3, followed by compiling old code with
>>> scons2. scons2 errors out with a pickling error. This is perfectly fine.
>>> The developer removes the .sconsign.dblite file and all is good.
>>> - Compile old code with scons2, followed by compiling new code with
>>> scon3. The build is successful, but the build is corrupt. *This is
>>> not acceptable.*
>>>
>>>
>>> We have unknowingly deployed bad builds to the field, and this is
>>> causing all sorts of havoc. I have added the below to our SConstruct file
>>> for one project. Can something like this be added to scons?
>>>
>>> Thanks,
>>>
>>> Jay
>>>
>>> def fix_scons_db():
>>> '''
>>> Fix scons2 vs scons3 DB backward compatibility problem
>>>
>>> The dependency database from scons-3.1.2 cannot be read by
>>> scons-4.3.0. Instead of generating an error, scons-4.3.0 will blissfully
>>> ignore the error and produce an incorrect build.
>>> In order to workaround this, we'll use use an scons
>>> version-specific DB file. Credit for the idea to:
>>>
>>> https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022
>>> This solution also future proofs against further scons upgrades.
>>> '''
>>> scons_db_file_name = '.sconsign.{}'.format(SCons.__version__)
>>> SConsignFile(scons_db_file_name)
>>> scons_db_file_name += '.dblite' # scons adds this suffix
>>> scons_old_db_files = glob.glob('.sconsign.*')
>>> if scons_db_file_name in scons_old_db_files:
>>> scons_old_db_files.remove(scons_db_file_name)
>>> #print('scons DB file name:', scons_db_file_name, 'removing',
>>> scons_old_db_files)
>>> if len(scons_old_db_files):
>>> subprocess.run(['rm -f .sconsign.*'], check=True, shell=True)
>>>
>>>
>>> On Sun, Mar 20, 2022 at 9:36 PM Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>>> There's no (reasonable) way to make the sconsign files be compatible
>>>> between py3.5 and p2.7.
>>>> The only way to not have invalid (as in crashes scons per your original
>>>> email) is to name the sconsign files per python version.
>>>> (Which I mentioned in my previous email, and is what the commit you
>>>> pointed to).
>>>>
>>>> Of course as you discovered (most likely) you will rebuild items which
>>>> were not out of date.
>>>>
>>>> The recommended way is to just validate the new scons & python
>>>> combination on your build and then make the switch.
>>>>
>>>> -Bill
>>>>
>>>>
>>>> On Sun, Mar 20, 2022 at 7:23 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>
>>>>> Actually, the workaround isn't sufficient. We also need to delete all
>>>>> the other versions of .sconsign.dblite* as they will be out of date.
>>>>>
>>>>> Jay
>>>>>
>>>>>
>>>>> On Sun, Mar 20, 2022 at 6:58 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>>
>>>>>> If scons did a full rebuild, it would not be a problem. The issue is
>>>>>> that scons does NOT do a full rebuild, thinking that targets were up to
>>>>>> date when they were really not.
>>>>>>
>>>>>> I see a workaround that someone else took:
>>>>>> https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022.
>>>>>> I will be putting this into our SContruct file. Can this be made the
>>>>>> default behavior?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jay
>>>>>>
>>>>>>
>>>>>> On Sat, Mar 19, 2022 at 11:09 AM Bill Deegan <
>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>
>>>>>>> Switching from py2.7 -> py 3.5 invalidates the .sconsign files
>>>>>>> contents.
>>>>>>> You have to do a full build when you make that switch.
>>>>>>> It's in the release notes somewhere.
>>>>>>>
>>>>>>> https://scons.org/scons-300-is-available.html
>>>>>>>
>>>>>>> There's no way around it.
>>>>>>> The best you can do is name your SConsign file to include the python
>>>>>>> version so at least they won't be invalid when you switch from one to
>>>>>>> another, but you will see additional rebuilds which may not be strictly
>>>>>>> necessary.
>>>>>>>
>>>>>>> -Bill
>>>>>>>
>>>>>>> On Fri, Mar 18, 2022 at 5:03 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>>>>
>>>>>>>> Yes, our scons-3.1.2 was using python-2.7.17, and our scons-4.3.1
>>>>>>>> is using python-3.9.10. We get a pickling error when trying to run
>>>>>>>> scons-3.1.2 after scons-4.3.1 which is good. Running scons-4.3.1 after
>>>>>>>> scons-3.1.2 runs without error, but results in bad builds, which is
>>>>>>>> terrible.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 18, 2022 at 3:04 PM Bill Deegan <
>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>
>>>>>>>>> Sounds like you also switched from python 2.7 to python 3.5 or
>>>>>>>>> higher?
>>>>>>>>>
>>>>>>>>> On Fri, Mar 18, 2022 at 2:07 PM Jay West <jhdub23 at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> We are migrating from scons-3.1.2 to scons-4.3.0 and have
>>>>>>>>>> encountered incorrect builds.
>>>>>>>>>>
>>>>>>>>>> After building with scons-3.1.2, changing some files, and then
>>>>>>>>>> running scons-4.3.0, it appears that scons-4.3.0 cannot read some
>>>>>>>>>> dependencies from the DB file. Instead of error'ing out or ignoring the DB
>>>>>>>>>> file, it happily generates an incorrect build. However, running:
>>>>>>>>>>
>>>>>>>>>> sconsign .sconsign.dblite
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> gives
>>>>>>>>>>
>>>>>>>>>> TypeError: '<' not supported between instances of 'str' and
>>>>>>>>>> 'bytes'
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Has anyone seen this issue?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Jay
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Scons-users mailing list
>>>>>>>>>> Scons-users at scons.org
>>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-users mailing list
>>>>>>>>> Scons-users at scons.org
>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20220401/f9d291d6/attachment-0001.htm>
More information about the Scons-users
mailing list