[Scons-users] Windows Subsystem for Linux (WSL) with SCons for cross-compiling

Thu Jun 20 22:57:05 EDT 2019

I managed to get this to build (I won't say 'work') by using something like
the following (some bits missing but hopefully the general intent/method is
clear enough):

... platform identification... (set lib_ext, cross, a few other variables
for path names including 'ThirdParty_Lib' in this example)
lib_ext = "dll"
cross = "wsl i686-w64-mingw32-"

tgtName = "test"
tgtPath = PurePath(env['LV_Dirs'].dataDir) / (tgtName + '.' + lib_ext)
CXX = cross + "g++"
OUT = tgtPath.as_posix().replace(' ', r'\ ') # Change spaces to "\ ".
as_posix() gives me '/' as a path separator.

# MyDirP is a similarly formatted relative path to the third-party library
files and includes.

MY_FLAGS += f" -I{MyDirP} -L{MyDirP} -l{ThirdParty_Lib}"

bld = SCons.Builder.Builder(action = f"{CXX} -o {OUT} {my_source_paths}
{MY_FLAGS}")
# Give the Builder a specific name since the paths are hardcoded.
env.Append(BUILDERS = {'XShLib_MyLib': bld})

built_lib = env.XShLib_MyLib(target = str(tgtPath), source =
my_source_filenames)

The obvious problem here is that I've avoided using either $TARGET or
$SOURCE in the build command, instead hardcoding the strings in a format
for WSL. The Builder then becomes useless for any other set of
targets/sources.

Naively, I then interpreted some of the responses below:
Bill Deegan: "To get scons to think it's on a linux system when it does
such on a windows system, but only for one Environment() would be likely
way more work than it's worth."
Jason Kenny: "What you are wanting to do is cross windows build on Linux."
(This I knew, but reading it prompted me to consider what I tried next)
as a suggestion that perhaps I could create a new construction Environment
and manipulate the platform variable.

Setting platform='posix' is discussed in this StackOverflow question/answer
<https://stackoverflow.com/questions/49232835/scons-does-not-find-file-it-should-build-itself>(by
Bill Deegan) where it turns out I probably shouldn't try this.
Indeed, following Mats Wichmann's comments regarding the following: "How
are you planning to tell the environment that it should prepend 'wsl' to
the build lines?  I imagine you would modify the relevant *COM variables -
once we know if it's finding any at all."
gave me initial hope - setting a simple project up (with just a few files
to compile a library, initially starting with Linux only and avoiding the
cross compiling (Linux host, Windows target)) and then using the following
SConstruct (thank you for the env.Dump() suggestion - much better than
PrettyPrint).

winEnv = Environment()
print(winEnv.Dump())

linuxEnv = Environment(platform = 'posix', tools=['gcc', 'gnulink', 'ar',
'g++'])

if False:
tgtName = 'test.dll'
else:
tgtName = 'test.so'

listToPrepend = ['CCCOM', 'CXXCOM', 'SHCXXCOM', 'SHCCCOM', 'LINKCOM',
'SHLINKCOM']
for x in listToPrepend:
linuxEnv[x] = 'wsl ' + linuxEnv[x]

print(linuxEnv.Dump())

sourceList_Lib = ['myFunction.cpp']
lib_Files = [File(x) for x in sourceList_Lib]

libOutput = linuxEnv.SharedLibrary(tgtName, lib_Files)
Alias('all', libOutput)

Default('all')

gives me the correct command line for the first step:

scons: done reading SConscript files.
scons: Building targets ...
wsl g++ -o myFunction.os -c -fPIC myFunction.cpp

but then

scons: *** [myFunction.os] The system cannot find the file specified

which I gather is related to pretending I'm on a posix system when I'm not
(via the SO Q/A above)
bdbaddog:  The issue is the file separators.. so with posix x\b\c = a file.
With platform = win32 x\b\c = directory x\b and file c.. By incorrectly
setting platform to posix you got a mix of the above and chaos ensued with
the dependencies.

The common theme in at least some of the replies in this thread seems to be
that I should install SCons on the WSL instance, and then call that from
Windows.
I'm not exactly sure how this would be done whilst retaining the
appropriate dependencies, but I can imagine that it would indeed solve the
problems with paths.
I think I will first try manipulating the $TARGET and $SOURCE values in the
Command (although I didn't have any luck with this yesterday), but that may
be my next option.

As a separate question, I found that if I write

built_lib = env.XShLib_MyLib(...)

env.Depends('myNotYetDefinedTargetThatDependsOnThisLib', built_lib)

it doesn't seem to create any dependency.
I think this might be related to the issue described by garyo at
https://github.com/scons/scons/wiki/DependsAndAliases (which is somewhat
the opposite, but I think perhaps the same issue).
I tried using env.Depends(env.Alias('myNotYetDefinedTarget...'), built_lib)
but that also didn't work.

I managed to get around this by raising a specific exception which is then
caught and passed (the code being discussed in this thread is in a separate
file to the tool which is building most of my packages, and I can't invert
the ordering of execution), giving the "built_lib" as an argument, and then
placing that into a Depends call after defining the target, but I wondered
if there's a more appropriate way to use the Depends function when it's
known that the target with a dependency hasn't been defined yet.
For clarity, here's the code that does this (r is a instantiation of a
class containing a bunch of information about paths and names of a library
to build with the tool, and this runs in a loop over a list of rs):

dependToAdd = None
if r.instructionsFile:

extraFile = r.instructionsFile

print_info(f"Executing the contents of the file {extraFile}")

try:

exec(open(extraFile).read())

except SkipLibrary as error:

print_info(str(error))

continue

except AddDepends as error:

dependToAdd = error.args

pass

# Do stuff to define the target package
tgt = env.MyTool(tgtPath, r.srcPath)
# Add aliases, blah blah
if dependToAdd:

print_info("Adding dependency on " + ", ".join(str(x) for x in dependToAdd))
env.Depends(tgt, dependToAdd)

On Fri, 21 Jun 2019 at 05:07, Bill Deegan <bill at baddogconsulting.com> wrote:

> I'd argue this is not really a cross compile.
>
> wsl <command>
> is really not different than
> ssh user at host -c "<command>"
>
> Which would be unwise (in almost every case) to do in the case the
> filesystem isn't shared.
>
>
> On Thu, Jun 20, 2019 at 9:03 AM Jason Kenny <dragon512 at live.com> wrote:
>
>> What you are wanting to do is cross windows build on Linux. That is what
>> the x86_64-w64-mingw32-g++ tool is doing for you. You happen to be doing
>> a cross windows build on linux on a windows box via the WSL.
>>
>> there is two way for you to go forward on this.
>>
>> 1) install python/SCons on the WSL and use it  ( do a window cross build
>> on linux )
>> 2) install Mingw on windows and use it with a windows install of python
>> and scons  (do a native windows build with gcc)
>>
>> I think part of the confusion is that the WSL is a sub-system on windows
>> ( ie a system within a system that does not require a VM to run). The only
>> real difference with WSL1 and WSL2 will be that we will have a full Linux
>> kernel running and mapped to the windows kernel, vs partial emulation. This
>> will allow the user to run many different versions of Linux on windows at
>> the same time. Hopefully, the rest of the issues I have seen such as
>> network/port emulation on /dev will be fixed.  This does not make a windows
>> box a Linux system. When running under the WSL you have to install
>> everything you want the WSL instance to use. windows tools do not work in
>> the wsl and Linux tools only see the windows file system through what is
>> basically a samba mount
>>
>> You are having Scons invoke the default wsl instance to do a command
>> (such as GCC ) from within the WSL. This is really no different than having
>> SCons call gcc through docker. The main issue here is the scons does not
>> see what is in the docker image on the windows side ( same with WSL).
>> Because of this, it cannot configure itself correctly
>>
>> Jason
>> ------------------------------
>> *From:* Scons-users <scons-users-bounces at scons.org> on behalf of
>> damien at khubla.com <damien at khubla.com>
>> *Sent:* Thursday, June 20, 2019 10:28 AM
>> *To:* 'SCons users mailing list'
>> *Subject:* Re: [Scons-users] Windows Subsystem for Linux (WSL) with
>> SCons for cross-compiling
>>
>> -----Original Message-----
>> From: Scons-users <scons-users-bounces at scons.org> On Behalf Of Mats
>> Wichmann
>> Sent: Thursday, June 20, 2019 9:18 AM
>> To: SCons users mailing list <scons-users at scons.org>; Bill Deegan <
>> bill at baddogconsulting.com>
>> Doesn't WSL2 fix this with a real Linux subsystem all the way down? We
>> ran into the same problems with WSL 1, but we do actual Linux builds anyway
>> so it didn't really impact us, it was just for convenience.
>>
>> Damien
>>
>> Subject: Re: [Scons-users] Windows Subsystem for Linux (WSL) with SCons
>> for cross-compiling
>>
>> On 6/20/19 8:56 AM, Bill Deegan wrote:
>> > I"m going to agree with Jason.
>> ...
>> > SCons interrogates the system it's on to decide about initializing
>> > various tools. To get scons to think it's on a linux system when it
>> > does such on a windows system, but only for one Environment() would be
>> > likely way more work than it's worth.
>> >
>> > BTW - Mats - I wouldn't consider this really cross building as it's
>> > really building on linux from windows.  And it's not that bad to setup.
>> > Of course pull request to make it easier to cross compile and/or
>> > docs/wiki pages always welcome.. ;)
>>
>> Well, it kinda is:  scons running in the windows context to run commands
>> which are running in the linux context.  But yeah, it's not the classical
>> case, which I don't know how to fix or I would have done it for IoTivity
>> (of course I knew much much less about scons then!).
>>
>> But I think this was really the crux of it: if it's Windows scons,
>> everything it finds is going to be in the Windows filesystem, then you try
>> to run actions which are going to need tools in the Linux container
>> - and last I knew Microsoft still said there wasn't support for Windows
>> commands looking at files inside the Linux environment (although it works
>> the other direction).
>>
>> So scons will have detected all the wrong stuff.
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>>
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist4.pair.net%2Fmailman%2Flistinfo%2Fscons-users&data=02%7C01%7C%7C2e81a6dc5190499a38ba08d6f5940b61%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636966413524642342&sdata=NFSvrknBedkPe1YS4dNrhPMU7VbpbykKdRluFpmXMr4%3D&reserved=0
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>>
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist4.pair.net%2Fmailman%2Flistinfo%2Fscons-users&data=02%7C01%7C%7C2e81a6dc5190499a38ba08d6f5940b61%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636966413524642342&sdata=NFSvrknBedkPe1YS4dNrhPMU7VbpbykKdRluFpmXMr4%3D&reserved=0
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20190621/95858379/attachment-0001.html>