[Scons-users] Windows Subsystem for Linux (WSL) with SCons for cross-compiling

Fri Jun 21 12:05:14 EDT 2019

I haven't been following this in detail, but I would *definitely *suggest
just installing SCons on the WSL side. You can share the
SConstruct/SConscripts (keep the whole tree in the Windows filesystem, not
WSL of course) and adjust the flags as needed per OS, but your life will be
*much* easier if you run scons for windows on windows, and scons for linux
on linux (whether in WSL or any other linux). SCons does a bunch of stuff
internally to figure out where it's running and set things up for that OS.
If you do it this way it'll basically just work.

On Thu, Jun 20, 2019 at 10:57 PM Christian Butcher <chrisb2244 at gmail.com>
wrote:

> I managed to get this to build (I won't say 'work') by using something
> like the following (some bits missing but hopefully the general
> intent/method is clear enough):
>
> ... platform identification... (set lib_ext, cross, a few other variables
> for path names including 'ThirdParty_Lib' in this example)
> lib_ext = "dll"
> cross = "wsl i686-w64-mingw32-"
>
> tgtName = "test"
> tgtPath = PurePath(env['LV_Dirs'].dataDir) / (tgtName + '.' + lib_ext)
> CXX = cross + "g++"
> OUT = tgtPath.as_posix().replace(' ', r'\ ') # Change spaces to "\ ".
> as_posix() gives me '/' as a path separator.
>
> # MyDirP is a similarly formatted relative path to the third-party library
> files and includes.
>
> MY_FLAGS += f" -I{MyDirP} -L{MyDirP} -l{ThirdParty_Lib}"
>
> bld = SCons.Builder.Builder(action = f"{CXX} -o {OUT} {my_source_paths}
> {MY_FLAGS}")
> # Give the Builder a specific name since the paths are hardcoded.
> env.Append(BUILDERS = {'XShLib_MyLib': bld})
>
> built_lib = env.XShLib_MyLib(target = str(tgtPath), source =
> my_source_filenames)
>
>
> The obvious problem here is that I've avoided using either $TARGET or
> $SOURCE in the build command, instead hardcoding the strings in a format
> for WSL. The Builder then becomes useless for any other set of
> targets/sources.
>
> Naively, I then interpreted some of the responses below:
> Bill Deegan: "To get scons to think it's on a linux system when it does
> such on a windows system, but only for one Environment() would be likely
> way more work than it's worth."
> Jason Kenny: "What you are wanting to do is cross windows build on Linux."
> (This I knew, but reading it prompted me to consider what I tried next)
> as a suggestion that perhaps I could create a new construction Environment
> and manipulate the platform variable.
>
> Setting platform='posix' is discussed in this StackOverflow
> question/answer
> <https://stackoverflow.com/questions/49232835/scons-does-not-find-file-it-should-build-itself>(by
> Bill Deegan) where it turns out I probably shouldn't try this.
> Indeed, following Mats Wichmann's comments regarding the following: "How
> are you planning to tell the environment that it should prepend 'wsl' to
> the build lines?  I imagine you would modify the relevant *COM variables -
> once we know if it's finding any at all."
> gave me initial hope - setting a simple project up (with just a few files
> to compile a library, initially starting with Linux only and avoiding the
> cross compiling (Linux host, Windows target)) and then using the following
> SConstruct (thank you for the env.Dump() suggestion - much better than
> PrettyPrint).
>
> winEnv = Environment()
> print(winEnv.Dump())
>
> linuxEnv = Environment(platform = 'posix', tools=['gcc', 'gnulink', 'ar',
> 'g++'])
>
> if False:
> tgtName = 'test.dll'
> else:
> tgtName = 'test.so'
>
> listToPrepend = ['CCCOM', 'CXXCOM', 'SHCXXCOM', 'SHCCCOM', 'LINKCOM',
> 'SHLINKCOM']
> for x in listToPrepend:
> linuxEnv[x] = 'wsl ' + linuxEnv[x]
>
> print(linuxEnv.Dump())
>
> sourceList_Lib = ['myFunction.cpp']
> lib_Files = [File(x) for x in sourceList_Lib]
>
> libOutput = linuxEnv.SharedLibrary(tgtName, lib_Files)
> Alias('all', libOutput)
>
> Default('all')
>
>
> gives me the correct command line for the first step:
>
> scons: done reading SConscript files.
> scons: Building targets ...
> wsl g++ -o myFunction.os -c -fPIC myFunction.cpp
>
> but then
>
> scons: *** [myFunction.os] The system cannot find the file specified
>
> which I gather is related to pretending I'm on a posix system when I'm not
> (via the SO Q/A above)
> bdbaddog:  The issue is the file separators.. so with posix x\b\c = a
> file. With platform = win32 x\b\c = directory x\b and file c.. By
> incorrectly setting platform to posix you got a mix of the above and chaos
> ensued with the dependencies.
>
> The common theme in at least some of the replies in this thread seems to
> be that I should install SCons on the WSL instance, and then call that from
> Windows.
> I'm not exactly sure how this would be done whilst retaining the
> appropriate dependencies, but I can imagine that it would indeed solve the
> problems with paths.
> I think I will first try manipulating the $TARGET and $SOURCE values in
> the Command (although I didn't have any luck with this yesterday), but that
> may be my next option.
>
> As a separate question, I found that if I write
>
> built_lib = env.XShLib_MyLib(...)
>
> env.Depends('myNotYetDefinedTargetThatDependsOnThisLib', built_lib)
>
> it doesn't seem to create any dependency.
> I think this might be related to the issue described by garyo at
> https://github.com/scons/scons/wiki/DependsAndAliases (which is somewhat
> the opposite, but I think perhaps the same issue).
> I tried using env.Depends(env.Alias('myNotYetDefinedTarget...'),
> built_lib) but that also didn't work.
>
> I managed to get around this by raising a specific exception which is then
> caught and passed (the code being discussed in this thread is in a separate
> file to the tool which is building most of my packages, and I can't invert
> the ordering of execution), giving the "built_lib" as an argument, and then
> placing that into a Depends call after defining the target, but I wondered
> if there's a more appropriate way to use the Depends function when it's
> known that the target with a dependency hasn't been defined yet.
> For clarity, here's the code that does this (r is a instantiation of a
> class containing a bunch of information about paths and names of a library
> to build with the tool, and this runs in a loop over a list of rs):
>
> dependToAdd = None
> if r.instructionsFile:
>
> extraFile = r.instructionsFile
>
> print_info(f"Executing the contents of the file {extraFile}")
>
> try:
>
> exec(open(extraFile).read())
>
> except SkipLibrary as error:
>
> print_info(str(error))
>
> continue
>
> except AddDepends as error:
>
> dependToAdd = error.args
>
> pass
>
> # Do stuff to define the target package
> tgt = env.MyTool(tgtPath, r.srcPath)
> # Add aliases, blah blah
> if dependToAdd:
>
> print_info("Adding dependency on " + ", ".join(str(x) for x in
> dependToAdd))
> env.Depends(tgt, dependToAdd)
>
>
> On Fri, 21 Jun 2019 at 05:07, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> I'd argue this is not really a cross compile.
>>
>> wsl <command>
>> is really not different than
>> ssh user at host -c "<command>"
>>
>> Which would be unwise (in almost every case) to do in the case the
>> filesystem isn't shared.
>>
>>
>> On Thu, Jun 20, 2019 at 9:03 AM Jason Kenny <dragon512 at live.com> wrote:
>>
>>> What you are wanting to do is cross windows build on Linux. That is what
>>> the x86_64-w64-mingw32-g++ tool is doing for you. You happen to be
>>> doing a cross windows build on linux on a windows box via the WSL.
>>>
>>> there is two way for you to go forward on this.
>>>
>>> 1) install python/SCons on the WSL and use it  ( do a window cross build
>>> on linux )
>>> 2) install Mingw on windows and use it with a windows install of python
>>> and scons  (do a native windows build with gcc)
>>>
>>> I think part of the confusion is that the WSL is a sub-system on windows
>>> ( ie a system within a system that does not require a VM to run). The only
>>> real difference with WSL1 and WSL2 will be that we will have a full Linux
>>> kernel running and mapped to the windows kernel, vs partial emulation. This
>>> will allow the user to run many different versions of Linux on windows at
>>> the same time. Hopefully, the rest of the issues I have seen such as
>>> network/port emulation on /dev will be fixed.  This does not make a windows
>>> box a Linux system. When running under the WSL you have to install
>>> everything you want the WSL instance to use. windows tools do not work in
>>> the wsl and Linux tools only see the windows file system through what is
>>> basically a samba mount
>>>
>>> You are having Scons invoke the default wsl instance to do a command
>>> (such as GCC ) from within the WSL. This is really no different than having
>>> SCons call gcc through docker. The main issue here is the scons does not
>>> see what is in the docker image on the windows side ( same with WSL).
>>> Because of this, it cannot configure itself correctly
>>>
>>> Jason
>>> ------------------------------
>>> *From:* Scons-users <scons-users-bounces at scons.org> on behalf of
>>> damien at khubla.com <damien at khubla.com>
>>> *Sent:* Thursday, June 20, 2019 10:28 AM
>>> *To:* 'SCons users mailing list'
>>> *Subject:* Re: [Scons-users] Windows Subsystem for Linux (WSL) with
>>> SCons for cross-compiling
>>>
>>> -----Original Message-----
>>> From: Scons-users <scons-users-bounces at scons.org> On Behalf Of Mats
>>> Wichmann
>>> Sent: Thursday, June 20, 2019 9:18 AM
>>> To: SCons users mailing list <scons-users at scons.org>; Bill Deegan <
>>> bill at baddogconsulting.com>
>>> Doesn't WSL2 fix this with a real Linux subsystem all the way down? We
>>> ran into the same problems with WSL 1, but we do actual Linux builds anyway
>>> so it didn't really impact us, it was just for convenience.
>>>
>>> Damien
>>>
>>> Subject: Re: [Scons-users] Windows Subsystem for Linux (WSL) with SCons
>>> for cross-compiling
>>>
>>> On 6/20/19 8:56 AM, Bill Deegan wrote:
>>> > I"m going to agree with Jason.
>>> ...
>>> > SCons interrogates the system it's on to decide about initializing
>>> > various tools. To get scons to think it's on a linux system when it
>>> > does such on a windows system, but only for one Environment() would be
>>> > likely way more work than it's worth.
>>> >
>>> > BTW - Mats - I wouldn't consider this really cross building as it's
>>> > really building on linux from windows.  And it's not that bad to setup.
>>> > Of course pull request to make it easier to cross compile and/or
>>> > docs/wiki pages always welcome.. ;)
>>>
>>> Well, it kinda is:  scons running in the windows context to run commands
>>> which are running in the linux context.  But yeah, it's not the classical
>>> case, which I don't know how to fix or I would have done it for IoTivity
>>> (of course I knew much much less about scons then!).
>>>
>>> But I think this was really the crux of it: if it's Windows scons,
>>> everything it finds is going to be in the Windows filesystem, then you try
>>> to run actions which are going to need tools in the Linux container
>>> - and last I knew Microsoft still said there wasn't support for Windows
>>> commands looking at files inside the Linux environment (although it works
>>> the other direction).
>>>
>>> So scons will have detected all the wrong stuff.
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>>
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist4.pair.net%2Fmailman%2Flistinfo%2Fscons-users&data=02%7C01%7C%7C2e81a6dc5190499a38ba08d6f5940b61%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636966413524642342&sdata=NFSvrknBedkPe1YS4dNrhPMU7VbpbykKdRluFpmXMr4%3D&reserved=0
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>>
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist4.pair.net%2Fmailman%2Flistinfo%2Fscons-users&data=02%7C01%7C%7C2e81a6dc5190499a38ba08d6f5940b61%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636966413524642342&sdata=NFSvrknBedkPe1YS4dNrhPMU7VbpbykKdRluFpmXMr4%3D&reserved=0
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>

-- 
Gary
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20190621/da1a830d/attachment-0001.html>