[Scons-users] Using $SOURCE files with _concat and Command String Parsing

Christopher Murray murpup at gmail.com
Mon Aug 26 23:26:02 EDT 2013

This may not help entirely, but it may create the spark needed to push
through the issue. I did not attempt to parse in depth everything you
said, but you seem to be experiencing issues not entirely dissimilar to
what I ran into years ago when I was trying to get our mixed Fortran/C code
to compile under Cygwin using its native Python but using the Windows-based
MSVS/Intel Fortran tool chain. I had issues with getting the concatenation
process to work as needed. What I ended up doing was to define my own
concatenation routine and use it wherever _concat was not doing the job,
like this:

import build_utils # my own custom library of support functions

# Define our own concatenation routine so that path names can be
# permuted to use win32 path separators. It is in build_support.py
env['cvf_concat'] = build_utils.cvf_concat
env['_F90INCFLAGS'] = '${cvf_concat(INCPREFIX, F90PATH, INCSUFFIX, __env__,

Then in build_utils.py, there is the following (which, if I remember
correctly, is just a modified version of _concat):

from SCons.Subst import *
from SCons.PathList import *

# Redefine my own concatenation routine so that path names are treated
# correctly under cygwin.
def cvf_concat(prefix, list, suffix, env, f=lambda x: x, target=None,
"""Creates a new list from 'list' by first interpolating each
element in the list using the 'env' dictionary and then calling f
on the list, and finally concatenating 'prefix' and 'suffix' onto
each element of the list. A trailing space on 'prefix' or leading
space on 'suffix' will cause them to be put into separate list
elements rather than being concatenated."""

if not list:
return prefix

l = f(SCons.PathList.PathList(list).subst_path(env, target, source))

if l is not None:
list = l

result = []

# ensure that prefix and suffix are strings
prefix = str(env.subst(prefix, SCons.Subst.SUBST_RAW))
suffix = str(env.subst(suffix, SCons.Subst.SUBST_RAW))

for x in list:
if isinstance(x, SCons.Node.FS.File):
x = str(x)
x = PermutePath(x, "\\\\")
if x:
if prefix:
if prefix[-1] == ' ':
elif x[:len(prefix)] != prefix:
x = prefix + x


if suffix:
if suffix[0] == ' ':
elif x[-len(suffix):] != suffix:
result[-1] = result[-1]+suffix

return result

Also, I need to make 2 small changes to SCons itself to help with
interpreting the command line. The first of these may be relevant to your

1) In Subst.py, function add_to_current_word, we have to uncomment
the code related to handling of literals so it looks like this:

y = current_word + x

# We used to treat a word appended to a literal
# as a literal itself, but this caused problems
# with interpreting quotes around space-separated
# targets on command lines. Removing this makes
# none of the "substantive" end-to-end tests fail,
# so we'll take this out but leave it commented
# for now in case there's a problem not covered
# by the test cases and we need to resurrect this.
literal1 = self.literal(self[-1][-1])
literal2 = self.literal(x)
y = self.conv(y)
if is_String(y):
y = CmdStringHolder(y, literal1 or literal2)
#y = CmdStringHolder(y, None)
self[-1][-1] = y

(this code used to be part of SCons (from a patch I submitted), but the
developers commented it out in later versions for some reason - probably
because mixing use of MSVS/Cygwin is not a valid enough use case. Maybe
you can provide a more valid use case)

2) In FS.py, we have to add the following to the function
Otherwise, SCons won't correctly find needed .h files

elif sys.platform == 'cygwin':
return d.has_key(_my_normcase(name))

Hope that helps,

On Sun, Aug 25, 2013 at 1:08 AM, William D. Jones <thor0505 at comcast.net>wrote:

> Hello all,


> I am working on a moderate-sized project in my spare time and using SCons

> to manage the build. I am currently running SCons 2.2 and Python 2.7 on

> Microsoft Windows 7 64-bit Professional.


> My project requires use of an x86 assembler, C-compiler which understands

> Microsoft extensions and a specific linker capable of absolute

> code-positioning. To that end, I’m using the WATCOM suite of tools to build

> my source. I am aware that SCons does not have WATCOM support built-in, and

> so I have built my own rudimentary tools file for the time being. I intend

> to submit this as a patch in the near-future (but will ask for some

> guidelines beforehand). Thanks to the built-in environment variables that

> permit defining prefixes and suffixes for various options, the C compiler

> and assembler work without major issues.


> Unfortunately, the WATCOM linker, Wlink, has a rather esoteric command

> line syntax: all linker options must be preceded with a directive. One

> particular problem with this linker is that every file place on the command

> line must be preceded with the ‘file’ directive, or alternatively, have a

> trailing comma after each file.


> Assume I have 4 source files, called a.obj, b.obj, c.obj, and d.obj, and

> want to create a target e.exe. The correct way to link a file such as this

> is either of the following:


> wlink name e.exe option quiet file a.obj file b.obj file c.obj file d.obj

> wlink name e.exe option quiet file a.obj, b.obj, c.obj, d.obj


> I attempted to make sure SCons adds the file directive prefix by using the

> built-in _concat function, but concatenation


> env['LINK'] = 'wlink'

> #env['_LINKSOURCES'] = '${_concat("file ", SOURCES, LINKSUFFIX, __env__)}'

> #env['LINKPREFIX'] = 'file '

> #env['LINKSUFFIX'] = ''




> According to the SCons source code, SCons.Defaults, line 295 in

> _concat_ixes(), file nodes (of which $SOURCES is part) are deliberately

> skipped during _concatenation. I assume there is a logical reason for this,

> but without this, I cannot seem to create a command line which SCons will

> accept. That said, is there a means to prefix each source file with the

> appropriate ‘file ‘ directive without modifying SCons.



> Although Wlink doesn’t have a wonderful syntax, it also seems that SCons

> is modifying the command strings/arguments which are sent to the system,

> and subsequently incorrectly processed by Wlink. Without quotations around

> ‘$TARGET’, SCons doesn’t parse the $TARGET variable properly according to

> wlink, and wlink will complain about a nonexistent directory. Wlink itself

> ignores the quotes in that context, but SCons passes the quotes to the

> system. In addition, I have verified that Wlink will accept multiple files

> without trailing commas or preceding file directive using the following

> syntax:

> env['LINK'] = 'wlink'

> ...


> file{$SOURCES}"


> Using the example source files above, the new command line becomes:

> wlink name ‘e.exe’ option quiet file{a.obj b.obj c.obj d.obj"}


> However, for reasons I do not understand, SCons subtly changes the

> command-line so that trying to link files in this manner causes all but the

> first and last file (b.obj c.obj) to error out with ‘file not found’. This

> is consistent with not having quotes around the target with env[‘LINKCOM’].

> I believe that SCons is not setting the command line properly, because

> setting the ‘LINKCOM’ variable to the final command line directly, without

> using any other construction variables also results in a successful build

> (though of course this limits me to one possible executable :P):

> env['LINK'] = 'wlink'

> ...

> env['LINKCOM'] = "wlink name ‘e.exe’ option quiet file{a.obj b.obj c.obj

> d.obj"}


> I imagine I can figure out what is wrong by using debug=pdb, but I’m not

> sure how to set a breakpoint *just *before a command is executed so that

> I can see the true command line passed to the system.


> Due to the type of object files Wlink outputs, the need for compile for

> bare-metal hardware, and the absolute code-positing requirements, the

> linker cannot be replaced easily. But short of editing SCons itself, I do

> not see a method to get Wlink to behave properly. I am wondering if anyone

> can suggest a temporary solution that doesn’t require me to edit SCons, or

> if anyone else has experienced problems with command lines being parsed

> improperly?


> Thank you for your help in advance.


> Sincerely,


> --

> William D. Jones

> Rowan University | ECE | 2012

> Member IEEE

> Member Tau Beta Pi

> thor0505 at comcast.net

> Message sent using 'Windows Live Mail' client.


> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://four.pairlist.net/pipermail/scons-users/attachments/20130826/f38eeb50/attachment-0001.html

More information about the Scons-users mailing list