[Scons-users] SCons popularity
Ray Sheppard
rsheppar at iu.edu
Sun Jun 14 12:39:09 EDT 2015
Hi William,
Sorry, I guess it was late. I read a little about your StaticLibrary
builder and also know that SharedLibrary is a WinDoze only function. I
should not have picked a specific use case. All I meant was that if the
code is expecting one set of libraries, often I need to force it to read
an equivalent library with a different name/link than expected (a naming
example: think Atlas/LAPACK/MKL/LibSci etc). I am not sure there is an
easy way a user could ever change link objects. It sure would be nice,
but it isn't just a SCons issue. I was more soapboxing to the folks
that write the "shared" restriction into their builds. Sorry.
About the main subject, I understand that SCons needs to be very
adaptable to many environments. On top of that, developers might create
their own environment. What I meant was that to know the values of all
the variables (for example CCCOM and LIBPATH) which currently set,
gives a shorthand snapshot of what might be needed to be overridden on
the command line. "In the old days" this was done by a "dry run."
Later, they started calling it a "scan." When you were talking about
your append and replace switches it just struck me how nice to would be
to do a dump of them first so I could have a quick snapshot of the
current state of the world before I started monkeying with it. Thanks
for considering and responding to my comments.
Ray
On 6/14/2015 10:42 AM, William Blevins wrote:
>
> Ray,
>
> And I think you for your impressions, but it's hard for me to divine
> the desired behavior from the problem behavior so please excuse my
> excess questions.
>
> On Sun, Jun 14, 2015 at 3:10 AM, Ray Sheppard <rsheppar at iu.edu
> <mailto:rsheppar at iu.edu>> wrote:
>
> Hello again William,
> I am giving you my impressions of the product. I already said I
> am not a developer. I will address some of your comments.
> Today's supercomputers seldom permit serial code. All code is
> parallelized. Usage of 10's of thousands of ranks is not
> uncommon. The general build directory is a working scratch file
> system. Installations must put their executables in a different
> file system. Several systems do not even have home directory
> access during the execution of a job. While computer scientists
> seem to love shared libraries, in a modern system, they needlessly
> light up the network fabric in order to slow down execution.
> Often, shared expectations (as in *.so) must be accomplished as a
> a static import from a *.a library.
>
>
> I'm not sure if I understand the desired behavior for the
> shared/static comments.
>
> 1. Do you mean SCons building shared vs static?
> In theory, we could update the SCons LibraryAPI so something like
> SharedLibrary, StaticLibrary, Library( DEF_LINK_TYPE = static ); thus,
> exposing a default linktype variable, but this would be messy and
> probably wouldn't help much because you cannot dynamically set the
> LIBS list correctly. If the original code was intending to build
> SharedLibraries (IE. a bigger liblist), then switching to static might
> work, but the resulting files may be linking the same static archive
> several times... it would get really messy and would only work
> depending on the SConscript files were created. Too much maybe,
> could, possibly for me.
>
> 2. Or maybe you are talking about whether SCons chooses to link shared
> libraries over static libraries by default? I would be surprised if
> their isn't a switch for that, though I don't know what it is off the
> top of my head, or maybe its linker specific. If its not linker
> specific, then we could add a behavior switch, so that when variables
> can be added directly to the Default environment, this behavior could
> be control be controlled.
>
> Do other build systems support what are speaking of?
>
> This creates complicated compiling systems. Using a Cray as an
> example, the underlying compiler might be gcc/gfortran. However,
> the actual compiler has a name like ftn. It has a section for
> putting ftn specific flags (first section of the compile line) as
> well as native compiler flags (middle section of the compile
> line). Code which is looking for a clue, like mpicc to decide if
> it is an MPI code is out of luck because the compilers parse the
> code and auto link in its version of message passing as well as
> most common math libraries without specific links. Adding in
> unnecessary links (like -lmpi) only result in an aborted
> compilation. All of this requires modifications (from my
> experience) deep in the SCons specific files. When I use the term
> "head" I am referring to the framework's ability to divine its
> surroundings and make decisions. Your framework certainly does try
> to do that. I have read and even hard copied printed your user
> guide. Were I creating a project and trying to use SCons to write
> it, your user guide would be very helpful. If I have a package
> that thinks it has write permissions to /usr/local and needs to
> have specific flag structures, it is not so much.
>
>
> Is the /usr/local permission due to the SCons default install
> directory? SCons is python and python needs to write out compiled
> counterparts like pyc and pyo. You may be able to get around this
> with python specific switches:
> http://stackoverflow.com/questions/154443/how-to-avoid-pyc-files
>
> I do not wish to write my own builder or scanner. I just want to
> invoke a build with something like:
> scons all INSTALL_ROOT=/somewhere CC=something CFLAGS= "-x -y -z"
> FC=something_else FFLAGS="-a -b -c"
> Reading variables like LD_LIBRARY_PATH and CFLAGS rather than
> making up a new ones like LIBPATH (or at least aliasing one to
> another) would be a nice touch. I seem to have missed that
> section of the UG. Again, this is just the impression of someone
> who uses your product. You can do with these comments as you wish.
> Ray
>
>
> SCons supports more operating systems than just linux, so all
> variables will "probably" not be all the same for all platforms. If
> SCons only supported linux, then we wouldn't have these complaints.
> Developers need to lookup a mapping from native OS to SCons internal
> but it could be scripted making this a 1-time lookup ideally.
>
>
> On 6/14/2015 2:18 AM, William Blevins wrote:
>> Ray,
>>
>> I might have been mistaken about something which is rather
>> unfortunate. Apparently, commandline defines don't get placed
>> directly into the DefaultEnvironment. I would say this could be
>> done in 1-2 lines of code, but you will run into the issue of
>> which variable are replaced vs which are appended...
>>
>> Maybe we could come up with a scheme for doing this with all
>> internally supported variables...
>>
>> I'm a bit sad now,
>> William
>>
>> On Sun, Jun 14, 2015 at 1:49 AM, William Blevins
>> <wblevins001 at gmail.com <mailto:wblevins001 at gmail.com>> wrote:
>>
>> Ray,
>>
>> Variable examples are C/C++.
>>
>> On Sat, Jun 13, 2015 at 11:50 PM, Ray Sheppard
>> <rsheppar at iu.edu <mailto:rsheppar at iu.edu>> wrote:
>>
>> Hello William,
>> I only have experience with the SCons-based packages
>> handed to me. So possibly my issues are with the way
>> those were created and not what is possible. In general,
>> they assume that they know what directory structures to
>> expect and assume they have write privileges where they
>> don't.
>>
>>
>> I'm not sure how this is an issue with the tool; might be an
>> issue with where the code was located when executed, build
>> requirements failure, etc. I would expect general practice to
>> have file writes down the SConstruct directory tree, but this
>> is not required.
>>
>> In addition, they usually "see" a compiler that they
>> should not use and grab it.
>>
>>
>> SCons has to make some assumption or it wouldn't do
>> anything. Compiler can be overridden via variable like CCCOM.
>>
>> The fact that it is just a piece of a more complicated
>> compiler with multi-staged compiler flags goes right over
>> its head.
>>
>>
>> SCons doesn't have a head, but I assume you just want to add
>> compiler flags: CCFLAGS?
>>
>> In general, executables need to be static because
>> execution directories are thousands of clock cycles away
>> from either home directories or system libraries.
>>
>>
>> I'm not sure how this is an issue with the tool; might be an
>> issue with where the code was located when executed, build
>> requirements failure, etc. I would expect general practice to
>> have file writes down the SConstruct directory tree, but this
>> is not required.
>>
>> I have posed MPI questions here before that went unresolved.
>>
>>
>> SCons doesn't support clustering across machines natively;
>> not sure why else you might want messaging passing. There
>> might be a 3-rd party extension, but I haven't heard of one.
>>
>> There is a shared caching mechanism that might be helpful:
>> http://www.scons.org/doc/HTML/scons-user/ch24.html
>>
>> I am not a big fan of Auto-xxx/M4/Configure. I was happy
>> setting my flags and compilers in a Makefile and letting
>> it go.
>>
>>
>> Unfortunately, how SCons behaves is up to the build script
>> writer in many cases, but any environment variable can be
>> overridden on the SCons command line including common ones
>> like LIBPATH, CPPPATH, CCFLAGS, compilers, or any other
>> variable. Here is a bigger list of entries that SCons
>> supports: http://www.scons.org/doc/HTML/scons-user/apa.html
>>
>> There are cases where this will not work if the script writer
>> has explicitly env.Replaced the variable in question thus
>> overriding any cmdline inputs, but I'm not sure how that can
>> be helped other than via communicating build requirements.
>>
>> We can promote good practices, but we cannot enforced them.
>> I find that many developers learn a tool by learning just
>> enough to do "anything" and brute forcing the rest. I have
>> on many occasions scolded my own coworkers for not trying to
>> lookup the API before they "reinvented the wheel".
>>
>> However, a simple set of command line flags to override
>> SCons assumptions, similar to ./configure --yyy, is not
>> widely publicized.
>>
>>
>> The User Guide is very publicized.
>>
>> V/R,
>> William
>>
>> Ray
>>
>>
>> On 6/13/2015 11:28 PM, William Blevins wrote:
>>>
>>>
>>> On Sat, Jun 13, 2015 at 5:14 PM, Ray Sheppard
>>> <rsheppar at iu.edu <mailto:rsheppar at iu.edu>> wrote:
>>>
>>> Hi All,
>>> For what it is worth, without this functionality,
>>> use of SCons on supercomputers is problematic.
>>> Ray
>>>
>>> On 6/13/2015 10:44 AM, Vasily wrote:
>>>
>>>
>>> Yes, I was talking about those. I believe you
>>> can override their default values by passing
>>> them as arguments, like
>>> $ scons all INSTALL_ROOT=some-path
>>>
>>> That would need some testing, though - I did not
>>> try setting it to a directory *not* under
>>> SConstruct's directory.
>>>
>>> Thanks,
>>> Vasily
>>>
>>>
>>>
>>>
>>> It's not clear what you are implying.
>>>
>>> SCons supports setting variables on the command line,
>>> and SCons supports installing outside the SConstruct
>>> directory tree, and supercomputing has nothing to do
>>> with either of these items.
>>>
>>> V/R,
>>> William
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org <mailto:Scons-users at scons.org>
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org <mailto:Scons-users at scons.org>
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org <mailto:Scons-users at scons.org>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org <mailto:Scons-users at scons.org>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org <mailto:Scons-users at scons.org>
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150614/4f3c350f/attachment-0001.html>
More information about the Scons-users
mailing list