[Scons-users] conflict with scons.bat 3.0.0 and anaconda3

[Toby, Brian H.]

Here is a followup on the windows bug I previously reported with scons & Python 3. This is now documented as a Python bug (https://bugs.python.org/issue32457) and scons.bat will be patched in Anaconda, but I still think it is worthwhile fixing path usage in scons.bat.

Brian

Begin forwarded message:

From: Anaconda Enterprise Support <support at continuum.io<mailto:support at continuum.io>>
Subject: Re: scons error w/miniconda3 on Windows
Date: January 4, 2018 at 11:54:32 AM CST
To: <toby at anl.gov<mailto:toby at anl.gov>>
Reply-To: Anaconda Enterprise Support <support at anaconda.com<mailto:support at anaconda.com>>

__________________________________
Type your response ABOVE THIS LINE to reply
________________________________
Brian Toby
Subject: scons error w/miniconda3 on Windows
________________________________
JAN 04, 2018  |  11:53AM CST
Ally Elsey replied:

Hello,

We received an update from one of our Anaconda Distribution developers. He says that this is an upstream Python bug. He has submitted a bug to the official Python bug tracker, https://bugs.python.org/issue32457.

More from our developer:

The scons project should not be adding any . or .. entries to the Windows PATH env var as it will break things (even if Python implements a work-around).
You should avoid calling scons.bat for now. I have tried to contact the scons project to let them know about this issue but I have failed to reach them.
For now, I will build scons packages with a scons.bat that does not do this.

Please let me know if you have any other questions. I will let me know when there is a new version of the scans package in our Repo.

Best,

Ally Elsey
Enterprise Support
Anaconda Inc

DEC 28, 2017  |  06:18PM CST
Brian replied:
Thanks. FWIW, the error bit me a few days ago when scons crashed on me and I had to remember why and the workaround.
________________________________
DEC 26, 2017  |  05:20AM CST
Ally Elsey replied:

Hello,

I’m touching base with you because it has been awhile since the last update. I have raised an issue with our Anaconda Distribution team about this bug. I will let you know when I have any further updates. If you have any questions please just let me know.

Best,

Ally Elsey
Enterprise Support
Anaconda Inc

DEC 12, 2017  |  05:23AM CST
Ally Elsey replied:

Hello,

I would be happy to start this conversation with the Anaconda distribution team and see what they think. I will let you know as soon as I have an update.

Best,

Ally Elsey
Enterprise Support
Anaconda Inc

DEC 12, 2017  |  11:07PM CST
Brian replied:
Hi Ally,

Thanks for asking. I had decided not to push this issue, but since you ask, I see this as an unaddressed bug. It could be something in Python, but I suspect it is in the Anaconda environment management system. If I had more time I would try to prove that the problem shows up with anaconda Python 3.6 but not the generic release.

In any case, AFAIK, adding C:\conda3-64\Scripts\.. into a path (especially since C:\conda3-64 is already in the path) should not break anything. But right now, with Anaconda Python on Windows this definitely prevents Python from starting.

There are workarounds, but that does not address that this is a bug and if it is not in Anaconda’s code it should get into the Python bug tracker.

Also, I’d love some help on how to convince the scons folks to include a workaround into their code, but that may be a lost cause.

Brian
________________________________
DEC 11, 2017  |  10:49AM CST
Ally Elsey replied:

Hello,

I’m checking in because it has been a few days since the last update. Please let me know if you have any other questions!

Best,

Ally Elsey
Enterprise Support
Anaconda Inc

DEC 06, 2017  |  05:08PM CST
Jeremy Enns replied:

Brian,

I think the problem part is this C:\conda3-64\Scripts\.. the extra two periods after the backslash. It does not appear to be a valid windows path. So their batch script is adding an invalid location to your path. I think if you just REM out the path part of the batch file and allow it to use the path as it is, it may work for you.

I am not sure why they would have an issue with fixing their batch script but like we saw you can bypass the batch file part and call the python file directly.

Jeremy Enns

DEC 06, 2017  |  04:53PM CST
Brian replied:
Yes and the response has not been 100% positive, see below. They have said that in the 3.0.2 release they hope to switch back to creating a scons.exe file on install, so perhaps this change will be moot, but I’d like to convince them to tweak their .bat script so that it is compatible with Anaconda. Perhaps you folks can help with either of two things:

1) Why do I have problems with a indirect reference to a directory in my path when running Anaconda python 3.7 on windows? e.g. this path does not work:


C:\conda3-64\Scripts\;C:\conda3-64\Scripts\..;C:\conda3-64;…


but this nominally equivalent path does work:


C:\conda3-64\Scripts\;C:\conda3-64;C:\conda3-64;...
Can anyone explain why this happens?

2) Do you have anyone more conversant with cmd scripting who can create a more transparent command to get the parent directory than the command below or can explain what it does?

for %%i in ("%~dp0..") do set "folder=%%~fi"

Brian

Begin forwarded message:

From: Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com><mailto:bill at baddogconsulting.com>>
Subject: Re: [Scons-users] conflict with scons.bat 3.0.0 and anaconda3
Date: December 5, 2017 at 10:23:01 PM CST
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org><mailto:scons-users at scons.org>>
Reply-To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org><mailto:scons-users at scons.org>>

It would really be contrary to SCons policy to accept a change when we don't understand
a) why the existing file is broken
b) Why the change fixes it.

Are the conda folks any help in understanding either?
________________________________
DEC 06, 2017  |  04:53AM CST
Ally Elsey replied:

Hello,

Thanks for sharing this with us. I’m glad that you are able to workaround this issue for now. Have you also alerted the scons mailing list to this?

Best,

Ally Elsey
Enterprise Support
Anaconda Inc

DEC 06, 2017  |  10:20PM CST
Brian replied:
As an update on this bug, I see now exactly what is causing the problem though I don’t completely understand it, but do have a proposed fix that solves the problem at least for me. You might want to consider patching scons.bat until a new release addresses the bug.

The problem is that this statement:
set path=%~dp0;%~dp0..;%path%

is intended to include the directory where scons.bat is located and its parent into the path, but does this using relative addressing.

So my initial path starts with

C:\conda3-64;C:\conda3-64\Library\mingw-w64\bin;C:\conda3-64\Library\usr\bin;C:\conda3-64\Library\bin;C:\conda3-64\Scripts; (…)

When the path is updated I get

C:\conda3-64\Scripts\;C:\conda3-64\Scripts\..;C:\conda3-64;C:\conda3-64\Library\mingw-w64\bin;C:\conda3-64\Library\usr\bin;C:\conda3-64\Library\bin;C:\conda3-64\Scripts;

which somehow breaks Python. The second item in the path should be equivalent to the third, but is somehow breaking the Python module search. If I modify the .bat file replacing that line with these two lines, the 2nd entry in the path is C:\conda3-64; instead of C:\conda3-64\Scripts\..; (yes, repeated) and everything works fine.

for %%i in ("%~dp0..") do set "folder=%%~fi"
set path=%~dp0;%folder%;%path%

Brian
________________________________
DEC 04, 2017  |  12:29PM CST
Jeremy Enns replied:

Brian,

Thanks, just be aware that CCing support at anaconda.com<mailto:support at anaconda.com> will generate duplicate support tickets in our system. I will close the first dupe but please remove the email from future replies or it will create another ticket each time.

Jeremy Enns

DEC 04, 2017  |  12:17PM CST
Brian replied:
bug posted
________________________________
DEC 04, 2017  |  11:55AM CST
Brian replied:
I think subscription to a mailing list is needed (ugh!) see http://scons.org/lists.html unless they have moved to twitter (ugh**2).

Brian
________________________________
DEC 04, 2017  |  11:07AM CST
Jeremy Enns replied:

Brian,

I am not sure how to contact the developers of scons, their GitHub page has no issue tracker and I could not find any contact method in the wiki https://github.com/SCons/scons/wiki

Jeremy

DEC 04, 2017  |  10:52AM CST
Brian replied:
It appears that the .bat file prepends to the path. It is not clear to me why this would break Python, and the bug seems inconsistent, but I’d love to get some information to the scons developers so perhaps this can be fixed (by someone who knows .bat scripting better than I.) Any suggestions?

Brian
________________________________
DEC 01, 2017  |  05:40PM CST
Jeremy Enns replied:

Brian,

I think something funky is going on in the scons.bat script its rewriting the system path and might be causing something to break. One of the developers noted you can call the script directly and see the help output. Maybe its just the batch script that is broken and you can still use the package with python "C:\User\builder\Anaconda3\Scripts\scons.py" -h

Jeremy

DEC 01, 2017  |  05:18PM CST
Brian replied:
FWIW I believe that scons has worked for someone else in Windows and Python 3.6 (via anaconda). I did not get the error in 2.7.

Brian
________________________________
DEC 01, 2017  |  05:09PM CST
Jeremy Enns replied:

Brian,

We have never received any support requests around the usage of the scons package so I do not have a whole lot to go on as to why it would not find its own library files.

I created a new environment on my Windows 10 VM and get the same error message as you when trying to run the scons.bat file. I will check with the Anaconda team on what might be going on here. I suspect scons has to set some specific environment variables or something to find its own resources.

Jeremy

DEC 01, 2017  |  04:35PM CST
Original message
Brian wrote:

I installed miniconda (py3.6), updated and used conda install to load packages I need to do a build (scons, m2w64-gcc-fortran, numpy) but scons will not run with a funny error:

(root) C:\Users\toby.WIN10-VM\Documents>scons
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00001874 (most recent call first):

After this error, Python dies. Advice?

My small bit of browsing indicates this may be a setup issue, but I have done this on a machine that is pretty much a new install and have not changed any environment variables. I use activate to set up the path, etc.

conda list and DOS set command results are below. Will play with 2.7 now.

Brian

(root) C:\Users\toby.WIN10-VM>conda list
# packages in environment at c:\conda3-64:
#
asn1crypto 0.23.0 py36h546eea3_0
ca-certificates 2017.08.26 h94faf87_0
certifi 2017.11.5 py36hb8ac631_0
cffi 1.11.2 py36h829d1a5_0
chardet 3.0.4 py36h420ce6e_1
conda 4.3.30 py36h7e176b0_0
conda-env 2.6.0 h36134e3_1
console_shortcut 0.1.1 h6bb2dd7_3
cryptography 2.1.3 py36h0eac7a7_0
icc_rt 2017.0.4 h97af966_0
idna 2.6 py36h148d497_1
intel-openmp 2018.0.0 hd92c6cd_8
m2w64-binutils 2.25.1 5
m2w64-bzip2 1.0.6 6
m2w64-crt-git 5.0.0.4636.2595836 2
m2w64-gcc 5.3.0 6
m2w64-gcc-fortran 5.3.0 6
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-headers-git 5.0.0.4636.c0ad18a 2
m2w64-isl 0.16.1 2
m2w64-libiconv 1.14 6
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
m2w64-mpc 1.0.3 3
m2w64-mpfr 3.1.4 4
m2w64-windows-default-manifest 6.4 3
m2w64-winpthreads-git 5.0.0.4634.697f757 2
m2w64-zlib 1.2.8 10
menuinst 1.4.10 py36h42196fb_0
mkl 2018.0.1 h2108138_4
msys2-conda-epoch 20160418 1
numpy 1.13.3 py36ha320f96_0
openssl 1.0.2m h093b818_1
pip 9.0.1 py36h226ae91_4
pycosat 0.6.3 py36h413d8a4_0
pycparser 2.18 py36hd053e01_1
pyopenssl 17.4.0 py36h3316aad_0
pysocks 1.6.7 py36h698d350_1
python 3.6.3 h3b118a2_4
pywin32 221 py36h9c10281_0
requests 2.18.4 py36h4371aae_1
ruamel_yaml 0.11.14 py36h9b16331_2
scons 3.0.0 py36hb42adb4_0
setuptools 36.5.0 py36h65f9e6e_0
six 1.11.0 py36h4db2310_1
urllib3 1.22 py36h276f60a_0
vc 14 h2379b0c_2
vs2015_runtime 14.0.25123 hd4c4e62_2
wheel 0.30.0 py36h6c3ec14_1
win_inet_pton 1.0.1 py36he67d7fd_1
wincertstore 0.2 py36h7fe50ca_0
yaml 0.1.7 hc54c509_2

(root) C:\Users\toby.WIN10-VM>

(root) C:\Users\toby.WIN10-VM\Documents>set
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\toby.WIN10-VM\AppData\Roaming
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=WIN10-VM
ComSpec=C:\windows\system32\cmd.exe
CONDA_DEFAULT_ENV=root
CONDA_PREFIX=c:\conda3-64
CONDA_PS1_BACKUP=$P$G
DEFLOGDIR=C:\ProgramData\McAfee\DesktopProtection
FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer
FPS_BROWSER_USER_PROFILE_STRING=Default
HOMEDRIVE=C:
HOMEPATH=\Users\toby.WIN10-VM
LOCALAPPDATA=C:\Users\toby.WIN10-VM\AppData\Local
LOGONSERVER=\\WIN10-VM
NUMBER_OF_PROCESSORS=2
OS=Windows_NT
Path=c:\conda3-64;c:\conda3-64\Library\mingw-w64\bin;c:\conda3-64\Library\usr\bin;c:\conda3-64\Library\bin;c:\conda3-64\Scripts;C:\Program Files (x86)\Parallels\Parallels Tools\Applications;C:\ProgramData\Oracle\Java\javapath;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\Users\toby.WIN10-VM\AppData\Local\Microsoft\WindowsApps;
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=4501
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PROMPT=(root) $P$G
PSModulePath=C:\Program Files\WindowsPowerShell\Modules;C:\windows\system32\WindowsPowerShell\v1.0\Modules
PUBLIC=C:\Users\Public
SCONS_ERRORLEVEL=-1073740791
SESSIONNAME=Console
SystemDrive=C:
SystemRoot=C:\windows
TEMP=C:\Users\TOBY~1.WIN\AppData\Local\Temp
TMP=C:\Users\TOBY~1.WIN\AppData\Local\Temp
USERDOMAIN=WIN10-VM
USERDOMAIN_ROAMINGPROFILE=WIN10-VM
USERNAME=toby
USERPROFILE=C:\Users\toby.WIN10-VM
VSEDEFLOGDIR=C:\ProgramData\McAfee\DesktopProtection
windir=C:\windows

(root) C:\Users\toby.WIN10-VM\Documents>
































































































________________________________
This message was sent to toby at anl.gov<mailto:toby at anl.gov> in reference to Case #: 7972.
Follow this link to view the status of your case and add additional comments:
http://support.anaconda.com/customer/portal/private/cases/7972

*Important Note on Email Sent to Multiple Recipients
If you send or reply to a message with multiple recipients, any responses to the thread may show up as part of the case history, even if those exchanges aren't directed to you. In essence, the owner of the original message also owns all communication associated with that case, regardless of who the subsequent senders and recipients are. Our suggestion is to make sure that all recipients are aware of this, and that the sending of sensitive information is avoided.



[[0c46a80d5ef3af6f2a68a72aadb65c371d6d597c-1091606585]]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20180104/ac2b8001/attachment-0001.html>