[Scons-users] SCons popularity

Sun Jun 14 10:42:54 EDT 2015

Ray,

And I think you for your impressions, but it's hard for me to divine the
desired behavior from the problem behavior so please excuse my excess
questions.

On Sun, Jun 14, 2015 at 3:10 AM, Ray Sheppard <rsheppar at iu.edu> wrote:

>  Hello again William,
>   I am giving you my impressions of the product. I already said I am not a
> developer.  I will address some of your comments.  Today's supercomputers
> seldom permit serial code.  All code is parallelized. Usage of 10's of
> thousands of ranks is not uncommon.  The general build directory is a
> working scratch file system.  Installations must put their executables in a
> different file system.  Several systems do not even have home directory
> access during the execution of a job.  While computer scientists seem to
> love shared libraries, in a modern system, they needlessly light up the
> network fabric in order to slow down execution.  Often, shared expectations
> (as in *.so) must be accomplished as a a static import from a *.a library.
>

I'm not sure if I understand the desired behavior for the shared/static
comments.

1. Do you mean SCons building shared vs static?
In theory, we could update the SCons LibraryAPI so something like
SharedLibrary, StaticLibrary, Library( DEF_LINK_TYPE = static ); thus,
exposing a default linktype variable, but this would be messy and probably
wouldn't help much because you cannot dynamically set the LIBS list
correctly.  If the original code was intending to build SharedLibraries
(IE. a bigger liblist), then switching to static might work, but the
resulting files may be linking the same static archive several times... it
would get really messy and would only work depending on the SConscript
files were created.  Too much maybe, could, possibly for me.

2. Or maybe you are talking about whether SCons chooses to link shared
libraries over static libraries by default?  I would be surprised if their
isn't a switch for that, though I don't know what it is off the top of my
head, or maybe its linker specific.  If its not linker specific, then we
could add a behavior switch, so that when variables can be added directly
to the Default environment, this behavior could be control be controlled.

Do other build systems support what are speaking of?

> This creates complicated compiling systems.  Using a Cray as an example,
> the underlying compiler might be gcc/gfortran.  However, the actual
> compiler has a name like ftn.  It has a section for putting ftn specific
> flags (first section of the compile line) as well as native compiler flags
> (middle section of the compile line). Code which is looking for a clue,
> like mpicc to decide if it is an MPI code is out of luck because the
> compilers parse the code and auto link in its version of message passing as
> well as most common math libraries without specific links. Adding in
> unnecessary links (like -lmpi) only result in an aborted compilation. All
> of this requires modifications (from my experience) deep in the SCons
> specific files.  When I use the term "head" I am referring to the
> framework's ability to divine its surroundings and make decisions. Your
> framework certainly does try to do that. I have read and even hard copied
> printed your user guide. Were I creating a project and trying to use SCons
> to write it, your user guide would be very helpful. If I have a package
> that thinks it has write permissions to /usr/local and needs to have
> specific flag structures, it is not so much.
>

Is the /usr/local permission due to the SCons default install directory?
SCons is python and python needs to write out compiled counterparts like
pyc and pyo.  You may be able to get around this with python specific
switches: http://stackoverflow.com/questions/154443/how-to-avoid-pyc-files

> I do not wish to write my own builder or scanner.  I just want to invoke a
> build with something like:
> scons all INSTALL_ROOT=/somewhere CC=something CFLAGS= "-x -y -z"
> FC=something_else FFLAGS="-a -b -c"
> Reading variables like LD_LIBRARY_PATH and CFLAGS rather than making up a
> new ones like LIBPATH  (or at least aliasing one to another) would be a
> nice touch.  I seem to have missed that section of the UG. Again, this is
> just the impression of someone who uses your product.  You can do with
> these comments as you wish.
>                          Ray
>
>
SCons supports more operating systems than just linux, so all variables
will "probably" not be all the same for all platforms.  If SCons only
supported linux, then we wouldn't have these complaints.
Developers need to lookup a mapping from native OS to SCons internal but it
could be scripted making this a 1-time lookup ideally.

> On 6/14/2015 2:18 AM, William Blevins wrote:
>
>  Ray,
>
>  I might have been mistaken about something which is rather unfortunate.
> Apparently, commandline defines don't get placed directly into the
> DefaultEnvironment.  I would say this could be done in 1-2 lines of code,
> but you will run into the issue of which variable are replaced vs which are
> appended...
>
>  Maybe we could come up with a scheme for doing this with all internally
> supported variables...
>
>  I'm a bit sad now,
>  William
>
> On Sun, Jun 14, 2015 at 1:49 AM, William Blevins <wblevins001 at gmail.com>
> wrote:
>
>>   Ray,
>>
>>  Variable examples are C/C++.
>>
>> On Sat, Jun 13, 2015 at 11:50 PM, Ray Sheppard <rsheppar at iu.edu> wrote:
>>
>>>  Hello William,
>>>   I only have experience with the SCons-based packages handed to me. So
>>> possibly my issues are with the way those were created and not what is
>>> possible.  In general, they assume that they know what directory structures
>>> to expect and assume they have write privileges where they don't.
>>>
>>
>> I'm not sure how this is an issue with the tool; might be an issue with
>> where the code was located when executed, build requirements failure, etc.
>> I would expect general practice to have file writes down the SConstruct
>> directory tree, but this is not required.
>>
>>
>>> In addition, they usually "see" a compiler that they should not use and
>>> grab it.
>>>
>>
>>  SCons has to make some assumption or it wouldn't do anything.  Compiler
>> can be overridden via variable like CCCOM.
>>
>>
>>> The fact that it is just a piece of a more complicated compiler with
>>> multi-staged compiler flags goes right over its head.
>>>
>>
>>  SCons doesn't have a head, but I assume you just want to add compiler
>> flags: CCFLAGS?
>>
>>
>>> In general, executables need to be static because execution directories
>>> are thousands of clock cycles away from either home directories or system
>>> libraries.
>>>
>>
>>  I'm not sure how this is an issue with the tool; might be an issue with
>> where the code was located when executed, build requirements failure, etc.
>> I would expect general practice to have file writes down the SConstruct
>> directory tree, but this is not required.
>>
>>
>>> I have posed MPI questions here before that went unresolved.
>>>
>>
>>  SCons doesn't support clustering across machines natively; not sure why
>> else you might want messaging passing.  There might be a 3-rd party
>> extension, but I haven't heard of one.
>>
>>  There is a shared caching mechanism that might be helpful:
>> http://www.scons.org/doc/HTML/scons-user/ch24.html
>>
>>
>>> I am not a big fan of Auto-xxx/M4/Configure.  I was happy setting my
>>> flags and compilers in a Makefile and letting it go.
>>>
>>
>> Unfortunately, how SCons behaves is up to the build script writer in many
>> cases, but any environment variable can be overridden on the SCons command
>> line including common ones like LIBPATH, CPPPATH, CCFLAGS, compilers, or
>> any other variable.  Here is a bigger list of entries that SCons supports:
>> http://www.scons.org/doc/HTML/scons-user/apa.html
>>
>> There are cases where this will not work if the script writer has
>> explicitly env.Replaced the variable in question thus overriding any
>> cmdline inputs, but I'm not sure how that can be helped other than via
>> communicating build requirements.
>>
>> We can promote good practices, but we cannot enforced them.  I find that
>> many developers learn a tool by  learning just enough to do "anything" and
>> brute forcing the rest.  I have on many occasions scolded my own coworkers
>> for not trying to lookup the API before they "reinvented the wheel".
>>
>>  However, a simple set of command line flags to override SCons
>>> assumptions, similar to ./configure --yyy,  is not widely publicized.
>>>
>>
>>  The User Guide is very publicized.
>>
>>  V/R,
>>  William
>>
>>
>>>                                 Ray
>>>
>>>
>>> On 6/13/2015 11:28 PM, William Blevins wrote:
>>>
>>>
>>>
>>> On Sat, Jun 13, 2015 at 5:14 PM, Ray Sheppard <rsheppar at iu.edu> wrote:
>>>
>>>> Hi All,
>>>>   For what it is worth, without this functionality, use of SCons on
>>>> supercomputers is problematic.
>>>>           Ray
>>>>
>>>> On 6/13/2015 10:44 AM, Vasily wrote:
>>>>
>>>>>
>>>>> Yes, I was talking about those. I believe you can override their
>>>>> default values by passing them as arguments, like
>>>>> $ scons all INSTALL_ROOT=some-path
>>>>>
>>>>> That would need some testing, though - I did not try setting it to a
>>>>> directory *not* under SConstruct's directory.
>>>>>
>>>>> Thanks,
>>>>> Vasily
>>>>>
>>>>>
>>>>>
>>>>
>>>  It's not clear what you are implying.
>>>
>>> SCons supports setting variables on the command line, and SCons supports
>>> installing outside the SConstruct directory tree, and supercomputing has
>>> nothing to do with either of these items.
>>>
>>>  V/R,
>>>  William
>>>
>>>
>>>>  _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing listScons-users at scons.orghttps://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>
>
> _______________________________________________
> Scons-users mailing listScons-users at scons.orghttps://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150614/316486c1/attachment-0001.html>