[Scons-users] SCons 3.0.0.alpha.20170614 available on testpypi
Bill Deegan
bill at baddogconsulting.com
Fri Jul 21 18:38:19 EDT 2017
Tim,
Can you make a simple testcase for this?
It's something we should have covered in the SCons testsuite regardless.
Likely a unittest would cover it. (src/engine/SCons/Node/FSTests.py)
Though a systemtest might also be warranted. (tests under tests/)
In general, content signatures should be bytes and generated using bytes,
and scanning files should only happen on text files and should be String's
(regardless of char based(py2) or unicode(py3))
-Bill
On Fri, Jul 21, 2017 at 6:19 PM, Tim Jenness <tjenness at lsst.org> wrote:
>
> My patch doesn’t quite work. The problem is that the byte-based findall()
> returns bytes and downstream code expects strings. It needs to convert all
> the results to strings (which may fail of course).
>
> Now that I’ve thought about it a bit more I think the underlying problem
> is in engine/SCons/Node/FS.py around line 2630:
>
> 2608 def get_text_contents(self):
> 2609 """
> 2610 This attempts to figure out what the encoding of the text
> is
> 2611 based upon the BOM bytes, and then decodes the contents so
> that
> 2612 it's a valid python string.
> 2613 """
> 2614 contents = self.get_contents()
> 2615 # The behavior of various decode() methods and functions
> 2616 # w.r.t. the initial BOM bytes is different for different
> 2617 # encodings and/or Python versions. ('utf-8' does not
> strip
> 2618 # them, but has a 'utf-8-sig' which does; 'utf-16' seems to
> 2619 # strip them; etc.) Just sidestep all the complication by
> 2620 # explicitly stripping the BOM before we decode().
> 2621 if contents[:len(codecs.BOM_UTF8)] == codecs.BOM_UTF8:
> 2622 return contents[len(codecs.BOM_UTF8):].decode('utf-8')
> 2623 if contents[:len(codecs.BOM_UTF16_LE)] ==
> codecs.BOM_UTF16_LE:
> 2624 return contents[len(codecs.BOM_UTF16_
> LE):].decode('utf-16-le')
> 2625 if contents[:len(codecs.BOM_UTF16_BE)] ==
> codecs.BOM_UTF16_BE:
> 2626 return contents[len(codecs.BOM_UTF16_
> BE):].decode('utf-16-be')
> 2627 try:
> 2628 return contents.decode()
> 2629 except (UnicodeDecodeError, AttributeError) as e:
> 2630 return contents
>
> The problem is that if we fail to convert the bytes to Unicode the method
> returns the “text” contents in bytes. This breaks the contract of
> get_text_contents() promising to return a string.
>
> Removing the try block at line 2627 and instead using “return
> contents.decode(errors=“ignore”)” fixes my boost.python build problem.
>
> —
> Tim Jenness
>
>
>
> On Jul 21, 2017, at 11:36 , Tim Jenness <tjenness at lsst.org> wrote:
>
> I’ve been testing this alpha release on our codebase.
>
> So far I’ve only encountered one problem inside SCons. In
> Scanner/__init__.py
>
> When I use TryCompile() the header files are scanned and there is a
> problem if the header files can’t be converted to Unicode.
>
> This triggered for me in boost-python where some of the header files have
> a corrupt character in the author’s name (German double-s I think). This
> leads to get_text_contents() returning bytes instead of str which causes
> the regex to fail. The fix is pretty straightforward:
>
> Add
>
> self.creb = re.compile(regex.encode(), re.M)
>
> to have a bytes version of the regex, and then do:
>
> def find_include_names(self, node):
> text = node.get_text_contents()
> if isinstance(text, bytes):
> return self.creb.findall(text)
> return self.cre.findall(text)
>
> This allows me to build code that uses boost-python.
>
> —
> Tim Jenness
>
>
> On Jun 20, 2017, at 20:38 , Bill Deegan <bill at baddogconsulting.com> wrote:
>
> merged!
>
> Thanks!
>
> On Tue, Jun 20, 2017 at 5:03 PM, RW via Scons-users <scons-users at scons.org
> > wrote:
>
>> It turns out it was a quick fix for this one
>> https://bitbucket.org/scons/scons/pull-requests/482
>>
>> I also had a look around for anything similar that might be in the source
>> but didn't find anything
>>
>> On 20 June 2017 at 15:55, RW <garlicbready at googlemail.com> wrote:
>>
>>> At the moment they're not passing (At least on my machine). All tests
>>> that use dir_fixture are currently breaking under windows for 3.0. I'll
>>> post a test.log after I've finished running it
>>>
>>>
>>> On 20 Jun 2017 15:08, "William Blevins" <wblevins001 at gmail.com> wrote:
>>>
>>> Then how are the existing tests passing on Windows?
>>>
>>> On Jun 20, 2017 5:57 AM, "RW via Scons-users" <scons-users at scons.org>
>>> wrote:
>>>
>>>> Just to follow this looks like a windows only bug
>>>> Linux doesn't seem to have this problem
>>>>
>>>> On 20 June 2017 at 10:20, RW <garlicbready at googlemail.com> wrote:
>>>>
>>>>> I think I've spotted another bug
>>>>> I've recently tried to use fixtures in a test I'm working on (python27
>>>>> / win10)
>>>>> This should use a sub directory of "image" relative to the test
>>>>> calling it, but seems to fail
>>>>> ```
>>>>> test.dir_fixture('image')
>>>>> ```
>>>>>
>>>>> Using a test already in the sources that uses test.dir_fixture to
>>>>> demonstrate
>>>>> Runs okay under 2.5.1, but fails under 3.0 alpha
>>>>> ```
>>>>> python runtest.py test\packaging\convenience-fun
>>>>> ctions\convenience-functions.py
>>>>> ```
>>>>>
>>>>> Failure message
>>>>> ```
>>>>> (testenv) D:\SourceControl\GitRepos\scons>python runtest.py
>>>>> test\packaging\convenience-functions\convenience-functions.py
>>>>> 1/1 (100.00%) D:\\SourceControl\\GitRepos\\s
>>>>> cons\\testenv\\Scripts\\python.exe -tt test\packaging\convenience-fun
>>>>> ctions\convenience-functions.py
>>>>> Traceback (most recent call last):
>>>>> File "test\packaging\convenience-functions\convenience-functions.py",
>>>>> line 36, in <module>
>>>>> test.dir_fixture( "image" )
>>>>> File "D:\SourceControl\GitRepos\scons\QMTest\TestCmd.py", line
>>>>> 1325, in dir_fixture
>>>>> for entry in os.listdir(spath):
>>>>> WindowsError: [Error 3] The system cannot find the path specified:
>>>>> '\\SourceControl\\GitRepos\\scons\\test\\fixture\\image/*.*'
>>>>> ```
>>>>>
>>>>> If I use a absolute directory path within the call to test.dir_fixture
>>>>> then this seems to work okay
>>>>>
>>>>> Many Thanks
>>>>> Richard
>>>>>
>>>>> On 14 June 2017 at 21:49, Bill Deegan <bill at baddogconsulting.com>
>>>>> wrote:
>>>>>
>>>>>> Greetings,
>>>>>>
>>>>>> This is an alpha quality release.
>>>>>> It's the first version to support both python 3.5+, 2.7.x, and pypy.
>>>>>> There have been 701 commits since the 2.5.1 release.
>>>>>>
>>>>>> Please report any issues to the scons-users mailing list.
>>>>>> (See: http://scons.org/lists.html )
>>>>>>
>>>>>>
>>>>>> You can download via:
>>>>>> pip install -i https://testpypi.python.org/pypi scons
>>>>>>
>>>>>> Please do so in a virtualenv and not in your python distro.
>>>>>>
>>>>>> RELEASE 3.0.0.alpha.20170614 - Mon, 14 Jun 2017 12:23:56 -0400
>>>>>>
>>>>>> Please consult the RELEASE.txt file for a summary of changes since
>>>>>> the last
>>>>>> release and consult the CHANGES.txt file for complete a list of
>>>>>> changes
>>>>>> since last release. This announcement highlights only the important
>>>>>> changes.
>>>>>> Please note the following important changes since release 2.5.1:
>>>>>> *IT IS NOT READY FOR PRODUCTION USE*
>>>>>>
>>>>>> This is the initial release supporting both python 3.5+ and
>>>>>> 2.7.x and pypy
>>>>>> There are some important changes:
>>>>>>
>>>>>> - Any print statements must now use python 3 syntax of
>>>>>> "print()"
>>>>>> - All node content should be in bytes. This is the default in
>>>>>> python 2.7.x,
>>>>>> in Python 3 all strings are by default unicode. byte and/or
>>>>>> bytearray
>>>>>> should be used if you construct content for return by a
>>>>>> custom node type's
>>>>>> get_content() method.
>>>>>> - This is some (as yet unresolved issue) using Literal()'s in
>>>>>> some context with
>>>>>> Python 3
>>>>>> - pypy should be supported, please report any issues to the
>>>>>> user's mailing list.
>>>>>> - Currently if you switch back and forth between python 2.7.x
>>>>>> and 3.5+ you will
>>>>>> need to remove your sconsign file. This should be resolves
>>>>>> shortly, but
>>>>>> regardless switching between python 2.7.x and 3.5+ will not
>>>>>> use compatible
>>>>>> sconsigns and as such incremental builds should be expected
>>>>>> to rebuild
>>>>>> anything changed since the previous scons run with the same
>>>>>> version of python.
>>>>>> - It is likely that migrating from 2.5.1 -> 3.0.0 alpha will
>>>>>> cause rebuilds due
>>>>>> to the significant number of changes in the codebase.
>>>>>>
>>>>>> - Removed deprecated tools CVS, Perforce, BitKeeper, RCS,
>>>>>> SCCS, Subversion.
>>>>>> - Removed deprecated module SCons.Sig
>>>>>> - See CHANGES.txt for more details on other changes
>>>>>> - 3.0.0 should be slightly faster than 2.5.1. Changes yielded
>>>>>> a 15% speed up for
>>>>>> null incremental builds.
>>>>>> - Updated D language scanner support to latest: 2.071.1.
>>>>>> - python -m SCons should now run SCons if it's installed
>>>>>> PYTHONPATH
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170721/e92bf46a/attachment-0001.html>
More information about the Scons-users
mailing list