[Scons-users] SCons 3.0.0.alpha.20170614 available on testpypi
Tim Jenness
tjenness at lsst.org
Fri Jul 21 18:19:21 EDT 2017
My patch doesn’t quite work. The problem is that the byte-based findall() returns bytes and downstream code expects strings. It needs to convert all the results to strings (which may fail of course).
Now that I’ve thought about it a bit more I think the underlying problem is in engine/SCons/Node/FS.py around line 2630:
2608 def get_text_contents(self):
2609 """
2610 This attempts to figure out what the encoding of the text is
2611 based upon the BOM bytes, and then decodes the contents so that
2612 it's a valid python string.
2613 """
2614 contents = self.get_contents()
2615 # The behavior of various decode() methods and functions
2616 # w.r.t. the initial BOM bytes is different for different
2617 # encodings and/or Python versions. ('utf-8' does not strip
2618 # them, but has a 'utf-8-sig' which does; 'utf-16' seems to
2619 # strip them; etc.) Just sidestep all the complication by
2620 # explicitly stripping the BOM before we decode().
2621 if contents[:len(codecs.BOM_UTF8)] == codecs.BOM_UTF8:
2622 return contents[len(codecs.BOM_UTF8):].decode('utf-8')
2623 if contents[:len(codecs.BOM_UTF16_LE)] == codecs.BOM_UTF16_LE:
2624 return contents[len(codecs.BOM_UTF16_LE):].decode('utf-16-le')
2625 if contents[:len(codecs.BOM_UTF16_BE)] == codecs.BOM_UTF16_BE:
2626 return contents[len(codecs.BOM_UTF16_BE):].decode('utf-16-be')
2627 try:
2628 return contents.decode()
2629 except (UnicodeDecodeError, AttributeError) as e:
2630 return contents
The problem is that if we fail to convert the bytes to Unicode the method returns the “text” contents in bytes. This breaks the contract of get_text_contents() promising to return a string.
Removing the try block at line 2627 and instead using “return contents.decode(errors=“ignore”)” fixes my boost.python build problem.
—
Tim Jenness
> On Jul 21, 2017, at 11:36 , Tim Jenness <tjenness at lsst.org> wrote:
>
> I’ve been testing this alpha release on our codebase.
>
> So far I’ve only encountered one problem inside SCons. In Scanner/__init__.py
>
> When I use TryCompile() the header files are scanned and there is a problem if the header files can’t be converted to Unicode.
>
> This triggered for me in boost-python where some of the header files have a corrupt character in the author’s name (German double-s I think). This leads to get_text_contents() returning bytes instead of str which causes the regex to fail. The fix is pretty straightforward:
>
> Add
>
> self.creb = re.compile(regex.encode(), re.M)
>
> to have a bytes version of the regex, and then do:
>
> def find_include_names(self, node):
> text = node.get_text_contents()
> if isinstance(text, bytes):
> return self.creb.findall(text)
> return self.cre.findall(text)
>
> This allows me to build code that uses boost-python.
>
> —
> Tim Jenness
>
>
>> On Jun 20, 2017, at 20:38 , Bill Deegan <bill at baddogconsulting.com <mailto:bill at baddogconsulting.com>> wrote:
>>
>> merged!
>>
>> Thanks!
>>
>> On Tue, Jun 20, 2017 at 5:03 PM, RW via Scons-users <scons-users at scons.org <mailto:scons-users at scons.org>> wrote:
>> It turns out it was a quick fix for this one
>> https://bitbucket.org/scons/scons/pull-requests/482 <https://bitbucket.org/scons/scons/pull-requests/482>
>>
>> I also had a look around for anything similar that might be in the source but didn't find anything
>>
>> On 20 June 2017 at 15:55, RW <garlicbready at googlemail.com <mailto:garlicbready at googlemail.com>> wrote:
>> At the moment they're not passing (At least on my machine). All tests that use dir_fixture are currently breaking under windows for 3.0. I'll post a test.log after I've finished running it
>>
>>
>> On 20 Jun 2017 15:08, "William Blevins" <wblevins001 at gmail.com <mailto:wblevins001 at gmail.com>> wrote:
>> Then how are the existing tests passing on Windows?
>>
>> On Jun 20, 2017 5:57 AM, "RW via Scons-users" <scons-users at scons.org <mailto:scons-users at scons.org>> wrote:
>> Just to follow this looks like a windows only bug
>> Linux doesn't seem to have this problem
>>
>> On 20 June 2017 at 10:20, RW <garlicbready at googlemail.com <mailto:garlicbready at googlemail.com>> wrote:
>> I think I've spotted another bug
>> I've recently tried to use fixtures in a test I'm working on (python27 / win10)
>> This should use a sub directory of "image" relative to the test calling it, but seems to fail
>> ```
>> test.dir_fixture('image')
>> ```
>>
>> Using a test already in the sources that uses test.dir_fixture to demonstrate
>> Runs okay under 2.5.1, but fails under 3.0 alpha
>> ```
>> python runtest.py test\packaging\convenience-functions\convenience-functions.py
>> ```
>>
>> Failure message
>> ```
>> (testenv) D:\SourceControl\GitRepos\scons>python runtest.py test\packaging\convenience-functions\convenience-functions.py
>> 1/1 (100.00%) D:\\SourceControl\\GitRepos\\scons\\testenv\\Scripts\\python.exe -tt test\packaging\convenience-functions\convenience-functions.py
>> Traceback (most recent call last):
>> File "test\packaging\convenience-functions\convenience-functions.py", line 36, in <module>
>> test.dir_fixture( "image" )
>> File "D:\SourceControl\GitRepos\scons\QMTest\TestCmd.py", line 1325, in dir_fixture
>> for entry in os.listdir(spath):
>> WindowsError: [Error 3] The system cannot find the path specified: '\\SourceControl\\GitRepos\\scons\\test\\fixture\\image/*.*'
>> ```
>>
>> If I use a absolute directory path within the call to test.dir_fixture then this seems to work okay
>>
>> Many Thanks
>> Richard
>>
>> On 14 June 2017 at 21:49, Bill Deegan <bill at baddogconsulting.com <mailto:bill at baddogconsulting.com>> wrote:
>> Greetings,
>>
>> This is an alpha quality release.
>> It's the first version to support both python 3.5+, 2.7.x, and pypy.
>> There have been 701 commits since the 2.5.1 release.
>>
>> Please report any issues to the scons-users mailing list.
>> (See: http://scons.org/lists.html <http://scons.org/lists.html> )
>>
>>
>> You can download via:
>> pip install -i https://testpypi.python.org/pypi <https://testpypi.python.org/pypi> scons
>>
>> Please do so in a virtualenv and not in your python distro.
>>
>> RELEASE 3.0.0.alpha.20170614 - Mon, 14 Jun 2017 12:23:56 -0400
>>
>> Please consult the RELEASE.txt file for a summary of changes since the last
>> release and consult the CHANGES.txt file for complete a list of changes
>> since last release. This announcement highlights only the important
>> changes.
>> Please note the following important changes since release 2.5.1:
>> *IT IS NOT READY FOR PRODUCTION USE*
>>
>> This is the initial release supporting both python 3.5+ and 2.7.x and pypy
>> There are some important changes:
>>
>> - Any print statements must now use python 3 syntax of "print()"
>> - All node content should be in bytes. This is the default in python 2.7.x,
>> in Python 3 all strings are by default unicode. byte and/or bytearray
>> should be used if you construct content for return by a custom node type's
>> get_content() method.
>> - This is some (as yet unresolved issue) using Literal()'s in some context with
>> Python 3
>> - pypy should be supported, please report any issues to the user's mailing list.
>> - Currently if you switch back and forth between python 2.7.x and 3.5+ you will
>> need to remove your sconsign file. This should be resolves shortly, but
>> regardless switching between python 2.7.x and 3.5+ will not use compatible
>> sconsigns and as such incremental builds should be expected to rebuild
>> anything changed since the previous scons run with the same version of python.
>> - It is likely that migrating from 2.5.1 -> 3.0.0 alpha will cause rebuilds due
>> to the significant number of changes in the codebase.
>>
>> - Removed deprecated tools CVS, Perforce, BitKeeper, RCS, SCCS, Subversion.
>> - Removed deprecated module SCons.Sig
>> - See CHANGES.txt for more details on other changes
>> - 3.0.0 should be slightly faster than 2.5.1. Changes yielded a 15% speed up for
>> null incremental builds.
>> - Updated D language scanner support to latest: 2.071.1.
>> - python -m SCons should now run SCons if it's installed PYTHONPATH
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org <mailto:Scons-users at scons.org>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org <mailto:Scons-users at scons.org>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org <mailto:Scons-users at scons.org>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org <mailto:Scons-users at scons.org>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users <https://pairlist4.pair.net/mailman/listinfo/scons-users>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org <mailto:Scons-users at scons.org>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170721/58ad5fb0/attachment-0001.html>
More information about the Scons-users
mailing list