[Scons-users] Using $SOURCE files with _concat and Command String Parsing

[Christopher Murray]

This may not help entirely, but it may create the spark needed to push
through the issue.  I did not attempt to parse in depth everything you
said, but you seem to be experiencing issues not entirely dissimilar to
what I ran into years ago when I was trying to get our mixed Fortran/C code
to compile under Cygwin using its native Python but using the Windows-based
MSVS/Intel Fortran tool chain.  I had issues with getting the concatenation
process to work as needed.  What I ended up doing was to define my own
concatenation routine and use it wherever _concat was not doing the job,
like this:


import build_utils    #  my own custom library of support functions

# Define our own concatenation routine so that path names can be
# permuted to use win32 path separators.  It is in build_support.py
env['cvf_concat'] = build_utils.cvf_concat
env['_F90INCFLAGS'] = '${cvf_concat(INCPREFIX, F90PATH, INCSUFFIX, __env__,
RDirs)}'


Then in build_utils.py, there is the following (which, if I remember
correctly, is just a modified version of _concat):


from SCons.Subst import *
from SCons.PathList import *

# Redefine my own concatenation routine so that path names are treated
# correctly under cygwin.
def cvf_concat(prefix, list, suffix, env, f=lambda x: x, target=None,
source=None):
    """Creates a new list from 'list' by first interpolating each
    element in the list using the 'env' dictionary and then calling f
    on the list, and finally concatenating 'prefix' and 'suffix' onto
    each element of the list. A trailing space on 'prefix' or leading
    space on 'suffix' will cause them to be put into separate list
    elements rather than being concatenated."""

    if not list:
        return prefix

    l = f(SCons.PathList.PathList(list).subst_path(env, target, source))

    if l is not None:
        list = l

    result = []

    # ensure that prefix and suffix are strings
    prefix = str(env.subst(prefix, SCons.Subst.SUBST_RAW))
    suffix = str(env.subst(suffix, SCons.Subst.SUBST_RAW))

    for x in list:
        if isinstance(x, SCons.Node.FS.File):
            result.append(x)
            continue
        x = str(x)
        x = PermutePath(x, "\\\\")
        if x:
            if prefix:
                if prefix[-1] == ' ':
                    result.append(prefix[:-1])
                elif x[:len(prefix)] != prefix:
                    x = prefix + x

            result.append(x)

            if suffix:
                if suffix[0] == ' ':
                    result.append(suffix[1:])
                elif x[-len(suffix):] != suffix:
                    result[-1] = result[-1]+suffix

    return result


Also, I need to make 2 small changes to SCons itself to help with
interpreting the command line.  The first of these may be relevant to your
issues:

1) In Subst.py, function add_to_current_word, we have to uncomment
   the code related to handling of literals so it looks like this:

         y = current_word + x

         # We used to treat a word appended to a literal
         # as a literal itself, but this caused problems
         # with interpreting quotes around space-separated
         # targets on command lines.  Removing this makes
         # none of the "substantive" end-to-end tests fail,
         # so we'll take this out but leave it commented
         # for now in case there's a problem not covered
         # by the test cases and we need to resurrect this.
         literal1 = self.literal(self[-1][-1])
         literal2 = self.literal(x)
         y = self.conv(y)
         if is_String(y):
             y = CmdStringHolder(y, literal1 or literal2)
             #y = CmdStringHolder(y, None)
         self[-1][-1] = y

(this code used to be part of SCons (from a patch I submitted), but the
developers commented it out in later versions for some reason - probably
because mixing use of MSVS/Cygwin is not a valid enough use case.  Maybe
you can provide a more valid use case)

2) In FS.py, we have to add the following to the function
entry_exists_on_disk
   Otherwise, SCons won't correctly find needed .h files

        elif sys.platform == 'cygwin':
            return d.has_key(_my_normcase(name))


Hope that helps,
Chris





On Sun, Aug 25, 2013 at 1:08 AM, William D. Jones <thor0505 at comcast.net>wrote:


>   Hello all,

>

> I am working on a moderate-sized project in my spare time and using SCons

> to manage the build. I am currently running SCons 2.2 and Python 2.7 on

> Microsoft Windows 7 64-bit Professional.

>

> My project requires use of an x86 assembler, C-compiler which understands

> Microsoft extensions and a specific linker capable of absolute

> code-positioning. To that end, I’m using the WATCOM suite of tools to build

> my source. I am aware that SCons does not have WATCOM support built-in, and

> so I have built my own rudimentary tools file for the time being. I intend

> to submit this as a patch in the near-future (but will ask for some

> guidelines beforehand). Thanks to the built-in environment variables that

> permit defining prefixes and suffixes for various options, the C compiler

> and assembler work without major issues.

>

> Unfortunately, the WATCOM linker, Wlink, has a rather esoteric command

> line syntax: all linker options must be preceded with a directive. One

> particular problem with this linker is that every file place on the command

> line must be preceded with the ‘file’ directive, or alternatively, have a

> trailing comma after each file.

>

> Assume I have 4 source files, called a.obj, b.obj, c.obj, and d.obj, and

> want to create a target e.exe. The correct way to link a file such as this

> is either of the following:

>

> wlink name e.exe option quiet file a.obj file b.obj file c.obj file d.obj

> wlink name e.exe option quiet file a.obj, b.obj, c.obj, d.obj

>

> I attempted to make sure SCons adds the file directive prefix by using the

> built-in _concat function, but concatenation

>

> env['LINK'] = 'wlink'

> #env['_LINKSOURCES'] = '${_concat("file ", SOURCES, LINKSUFFIX, __env__)}'

> #env['LINKPREFIX'] = 'file '

> #env['LINKSUFFIX'] = ''

> env['LINKCOM'] = "$LINK name '$TARGET' $LINKFLAGS $_LIBDIRFLAGS $_LIBFLAGS

> _LINKSOURCES”

>

> According to the SCons source code, SCons.Defaults, line 295 in

> _concat_ixes(), file nodes (of which $SOURCES is part) are deliberately

> skipped during _concatenation. I assume there is a logical reason for this,

> but without this, I cannot seem to create a command line which SCons will

> accept. That said, is there a means to prefix each source file with the

> appropriate ‘file ‘ directive without modifying SCons.

>

>

> Although Wlink doesn’t have a wonderful syntax, it also seems that SCons

> is modifying the command strings/arguments which are sent to the system,

> and subsequently incorrectly processed by Wlink. Without quotations around

> ‘$TARGET’, SCons doesn’t parse the $TARGET variable properly according to

> wlink, and wlink will complain about a nonexistent directory. Wlink itself

> ignores the quotes in that context, but SCons passes the quotes to the

> system. In addition, I have verified that Wlink will accept multiple files

> without trailing commas or preceding file directive using the following

> syntax:

> env['LINK'] = 'wlink'

> ...

> env['LINKCOM'] = "$LINK name '$TARGET' $LINKFLAGS $_LIBDIRFLAGS $_LIBFLAGS

> file{$SOURCES}"

>

> Using the example source files above, the new command line becomes:

> wlink name ‘e.exe’ option quiet file{a.obj b.obj c.obj d.obj"}

>

> However, for reasons I do not understand, SCons subtly changes the

> command-line so that trying to link files in this manner causes all but the

> first and last file (b.obj c.obj) to error out with ‘file not found’. This

> is consistent with not having quotes around the target with env[‘LINKCOM’].

> I believe that SCons is not  setting the command line properly, because

> setting the ‘LINKCOM’ variable to the final command line directly, without

> using any other construction variables also results in a successful build

> (though of course this limits me to one possible executable :P):

> env['LINK'] = 'wlink'

> ...

> env['LINKCOM'] = "wlink name ‘e.exe’ option quiet file{a.obj b.obj c.obj

> d.obj"}

>

> I imagine I can figure out what is wrong by using debug=pdb, but I’m not

> sure how to set a breakpoint *just *before a command is executed so that

> I can see the true command line passed to the system.

>

> Due to the type of object files Wlink outputs, the need for compile for

> bare-metal hardware, and the absolute code-positing requirements, the

> linker cannot be replaced easily. But short of editing SCons itself, I do

> not see a method to get Wlink to behave properly. I am wondering if anyone

> can suggest a temporary solution that doesn’t require me to edit SCons, or

> if anyone else has experienced problems with command lines being parsed

> improperly?

>

> Thank you for your help in advance.

>

> Sincerely,

>

> --

> William D. Jones

> Rowan University | ECE | 2012

> Member IEEE

> Member Tau Beta Pi

> thor0505 at comcast.net

> Message sent using 'Windows Live Mail' client.

>

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

>

>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://four.pairlist.net/pipermail/scons-users/attachments/20130826/f38eeb50/attachment-0001.html