[Scons-users] SCons 3.0.0.alpha.20170614 available on testpypi

[Tim Jenness]

> On Jul 21, 2017, at 15:19 , Tim Jenness <tjenness at lsst.org> wrote:
> 
> Now that I’ve thought about it a bit more I think the underlying problem is in engine/SCons/Node/FS.py around line 2630:
> 
>    2608     def get_text_contents(self):
>    2609         """
>    2610         This attempts to figure out what the encoding of the text is
>    2611         based upon the BOM bytes, and then decodes the contents so that
>    2612         it's a valid python string.
>    2613         """
>    2614         contents = self.get_contents()
>    2615         # The behavior of various decode() methods and functions
>    2616         # w.r.t. the initial BOM bytes is different for different
>    2617         # encodings and/or Python versions.  ('utf-8' does not strip
>    2618         # them, but has a 'utf-8-sig' which does; 'utf-16' seems to
>    2619         # strip them; etc.)  Just sidestep all the complication by
>    2620         # explicitly stripping the BOM before we decode().
>    2621         if contents[:len(codecs.BOM_UTF8)] == codecs.BOM_UTF8:
>    2622             return contents[len(codecs.BOM_UTF8):].decode('utf-8')
>    2623         if contents[:len(codecs.BOM_UTF16_LE)] == codecs.BOM_UTF16_LE:
>    2624             return contents[len(codecs.BOM_UTF16_LE):].decode('utf-16-le')
>    2625         if contents[:len(codecs.BOM_UTF16_BE)] == codecs.BOM_UTF16_BE:
>    2626             return contents[len(codecs.BOM_UTF16_BE):].decode('utf-16-be')
>    2627         try:
>    2628             return contents.decode()
>    2629         except (UnicodeDecodeError, AttributeError) as e:
>    2630             return contents
> 
> The problem is that if we fail to convert the bytes to Unicode the method returns the “text” contents in bytes. This breaks the contract of get_text_contents() promising to return a string.
> 
> Removing the try block at line 2627 and instead using “return contents.decode(errors=“ignore”)” fixes my boost.python build problem.
> 

This problem still exists at https://bitbucket.org/scons/scons/src/cfbc036995c8669e296cc0427655345241a0097e/src/engine/SCons/Node/FS.py?at=default&fileviewer=file-view-default#FS.py-2630 <https://bitbucket.org/scons/scons/src/cfbc036995c8669e296cc0427655345241a0097e/src/engine/SCons/Node/FS.py?at=default&fileviewer=file-view-default#FS.py-2630>

Should I be discussing this on the dev list? Sorry, but I don’t have time to learn how to do a bitbucket PR using hg. The good news is our 500,000 lines of python and C++ code built fine with my suggested fix in place.

Patch is at https://github.com/lsst/scons/blob/tickets/DM-8560/patches/0001-always-decode-contents.patch

— 
Tim Jenness

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170731/39a184e3/attachment.html>