[Scons-users] SCons popularity
William Blevins
wblevins001 at gmail.com
Sun Jun 14 12:52:46 EDT 2015
On Sun, Jun 14, 2015 at 12:49 PM, William Blevins <wblevins001 at gmail.com>
wrote:
>
>
> On Sun, Jun 14, 2015 at 12:39 PM, Ray Sheppard <rsheppar at iu.edu> wrote:
>
>> Hi William,
>> Sorry, I guess it was late. I read a little about your StaticLibrary
>> builder and also know that SharedLibrary is a WinDoze only function.
>>
>
> Not a WinDoze only function :)
>
>
>> I should not have picked a specific use case. All I meant was that if
>> the code is expecting one set of libraries, often I need to force it to
>> read an equivalent library with a different name/link than expected (a
>> naming example: think Atlas/LAPACK/MKL/LibSci etc). I am not sure there is
>> an easy way a user could ever change link objects. It sure would be nice,
>> but it isn't just a SCons issue. I was more soapboxing to the folks that
>> write the "shared" restriction into their builds. Sorry.
>> About the main subject, I understand that SCons needs to be very
>> adaptable to many environments. On top of that, developers might create
>> their own environment. What I meant was that to know the values of all the
>> variables (for example CCCOM and LIBPATH) which currently set, gives a
>> shorthand snapshot of what might be needed to be overridden on the command
>> line. "In the old days" this was done by a "dry run." Later, they started
>> calling it a "scan." When you were talking about your append and replace
>> switches it just struck me how nice to would be to do a dump of them first
>> so I could have a quick snapshot of the current state of the world before I
>> started monkeying with it.
>>
>
> You can indeed dump the environment:
> http://www.scons.org/doc/HTML/scons-user/ch28s02.html
>
Food for thought: we could add a debug switch that perform Dump on the
environment for all listed targets. That would more useful I believe.
>
>
>> Thanks for considering and responding to my comments.
>> Ray
>>
>>
>>
>
>>
>> On 6/14/2015 10:42 AM, William Blevins wrote:
>>
>>
>> Ray,
>>
>> And I think you for your impressions, but it's hard for me to divine the
>> desired behavior from the problem behavior so please excuse my excess
>> questions.
>>
>> On Sun, Jun 14, 2015 at 3:10 AM, Ray Sheppard <rsheppar at iu.edu> wrote:
>>
>>> Hello again William,
>>> I am giving you my impressions of the product. I already said I am not
>>> a developer. I will address some of your comments. Today's supercomputers
>>> seldom permit serial code. All code is parallelized. Usage of 10's of
>>> thousands of ranks is not uncommon. The general build directory is a
>>> working scratch file system. Installations must put their executables in a
>>> different file system. Several systems do not even have home directory
>>> access during the execution of a job. While computer scientists seem to
>>> love shared libraries, in a modern system, they needlessly light up the
>>> network fabric in order to slow down execution. Often, shared expectations
>>> (as in *.so) must be accomplished as a a static import from a *.a library.
>>>
>>
>> I'm not sure if I understand the desired behavior for the shared/static
>> comments.
>>
>> 1. Do you mean SCons building shared vs static?
>> In theory, we could update the SCons LibraryAPI so something like
>> SharedLibrary, StaticLibrary, Library( DEF_LINK_TYPE = static ); thus,
>> exposing a default linktype variable, but this would be messy and probably
>> wouldn't help much because you cannot dynamically set the LIBS list
>> correctly. If the original code was intending to build SharedLibraries
>> (IE. a bigger liblist), then switching to static might work, but the
>> resulting files may be linking the same static archive several times... it
>> would get really messy and would only work depending on the SConscript
>> files were created. Too much maybe, could, possibly for me.
>>
>> 2. Or maybe you are talking about whether SCons chooses to link shared
>> libraries over static libraries by default? I would be surprised if their
>> isn't a switch for that, though I don't know what it is off the top of my
>> head, or maybe its linker specific. If its not linker specific, then we
>> could add a behavior switch, so that when variables can be added directly
>> to the Default environment, this behavior could be control be controlled.
>>
>> Do other build systems support what are speaking of?
>>
>>
>>> This creates complicated compiling systems. Using a Cray as an
>>> example, the underlying compiler might be gcc/gfortran. However, the
>>> actual compiler has a name like ftn. It has a section for putting ftn
>>> specific flags (first section of the compile line) as well as native
>>> compiler flags (middle section of the compile line). Code which is looking
>>> for a clue, like mpicc to decide if it is an MPI code is out of luck
>>> because the compilers parse the code and auto link in its version of
>>> message passing as well as most common math libraries without specific
>>> links. Adding in unnecessary links (like -lmpi) only result in an aborted
>>> compilation. All of this requires modifications (from my experience) deep
>>> in the SCons specific files. When I use the term "head" I am referring to
>>> the framework's ability to divine its surroundings and make decisions. Your
>>> framework certainly does try to do that. I have read and even hard copied
>>> printed your user guide. Were I creating a project and trying to use SCons
>>> to write it, your user guide would be very helpful. If I have a package
>>> that thinks it has write permissions to /usr/local and needs to have
>>> specific flag structures, it is not so much.
>>>
>>
>> Is the /usr/local permission due to the SCons default install
>> directory? SCons is python and python needs to write out compiled
>> counterparts like pyc and pyo. You may be able to get around this with
>> python specific switches:
>> http://stackoverflow.com/questions/154443/how-to-avoid-pyc-files
>>
>>
>>> I do not wish to write my own builder or scanner. I just want to invoke
>>> a build with something like:
>>> scons all INSTALL_ROOT=/somewhere CC=something CFLAGS= "-x -y -z"
>>> FC=something_else FFLAGS="-a -b -c"
>>> Reading variables like LD_LIBRARY_PATH and CFLAGS rather than making up
>>> a new ones like LIBPATH (or at least aliasing one to another) would be a
>>> nice touch. I seem to have missed that section of the UG. Again, this is
>>> just the impression of someone who uses your product. You can do with
>>> these comments as you wish.
>>> Ray
>>>
>>>
>> SCons supports more operating systems than just linux, so all variables
>> will "probably" not be all the same for all platforms. If SCons only
>> supported linux, then we wouldn't have these complaints.
>> Developers need to lookup a mapping from native OS to SCons internal
>> but it could be scripted making this a 1-time lookup ideally.
>>
>>
>>> On 6/14/2015 2:18 AM, William Blevins wrote:
>>>
>>> Ray,
>>>
>>> I might have been mistaken about something which is rather
>>> unfortunate. Apparently, commandline defines don't get placed directly
>>> into the DefaultEnvironment. I would say this could be done in 1-2 lines
>>> of code, but you will run into the issue of which variable are replaced vs
>>> which are appended...
>>>
>>> Maybe we could come up with a scheme for doing this with all
>>> internally supported variables...
>>>
>>> I'm a bit sad now,
>>> William
>>>
>>> On Sun, Jun 14, 2015 at 1:49 AM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>>> Ray,
>>>>
>>>> Variable examples are C/C++.
>>>>
>>>> On Sat, Jun 13, 2015 at 11:50 PM, Ray Sheppard <rsheppar at iu.edu> wrote:
>>>>
>>>>> Hello William,
>>>>> I only have experience with the SCons-based packages handed to me.
>>>>> So possibly my issues are with the way those were created and not what is
>>>>> possible. In general, they assume that they know what directory structures
>>>>> to expect and assume they have write privileges where they don't.
>>>>>
>>>>
>>>> I'm not sure how this is an issue with the tool; might be an issue with
>>>> where the code was located when executed, build requirements failure, etc.
>>>> I would expect general practice to have file writes down the SConstruct
>>>> directory tree, but this is not required.
>>>>
>>>>
>>>>> In addition, they usually "see" a compiler that they should not use
>>>>> and grab it.
>>>>>
>>>>
>>>> SCons has to make some assumption or it wouldn't do anything.
>>>> Compiler can be overridden via variable like CCCOM.
>>>>
>>>>
>>>>> The fact that it is just a piece of a more complicated compiler with
>>>>> multi-staged compiler flags goes right over its head.
>>>>>
>>>>
>>>> SCons doesn't have a head, but I assume you just want to add compiler
>>>> flags: CCFLAGS?
>>>>
>>>>
>>>>> In general, executables need to be static because execution
>>>>> directories are thousands of clock cycles away from either home directories
>>>>> or system libraries.
>>>>>
>>>>
>>>> I'm not sure how this is an issue with the tool; might be an issue
>>>> with where the code was located when executed, build requirements failure,
>>>> etc. I would expect general practice to have file writes down the
>>>> SConstruct directory tree, but this is not required.
>>>>
>>>>
>>>>> I have posed MPI questions here before that went unresolved.
>>>>>
>>>>
>>>> SCons doesn't support clustering across machines natively; not sure
>>>> why else you might want messaging passing. There might be a 3-rd party
>>>> extension, but I haven't heard of one.
>>>>
>>>> There is a shared caching mechanism that might be helpful:
>>>> http://www.scons.org/doc/HTML/scons-user/ch24.html
>>>>
>>>>
>>>>> I am not a big fan of Auto-xxx/M4/Configure. I was happy setting my
>>>>> flags and compilers in a Makefile and letting it go.
>>>>>
>>>>
>>>> Unfortunately, how SCons behaves is up to the build script writer in
>>>> many cases, but any environment variable can be overridden on the SCons
>>>> command line including common ones like LIBPATH, CPPPATH, CCFLAGS,
>>>> compilers, or any other variable. Here is a bigger list of entries that
>>>> SCons supports: http://www.scons.org/doc/HTML/scons-user/apa.html
>>>>
>>>> There are cases where this will not work if the script writer has
>>>> explicitly env.Replaced the variable in question thus overriding any
>>>> cmdline inputs, but I'm not sure how that can be helped other than via
>>>> communicating build requirements.
>>>>
>>>> We can promote good practices, but we cannot enforced them. I find
>>>> that many developers learn a tool by learning just enough to do "anything"
>>>> and brute forcing the rest. I have on many occasions scolded my own
>>>> coworkers for not trying to lookup the API before they "reinvented the
>>>> wheel".
>>>>
>>>> However, a simple set of command line flags to override SCons
>>>>> assumptions, similar to ./configure --yyy, is not widely publicized.
>>>>>
>>>>
>>>> The User Guide is very publicized.
>>>>
>>>> V/R,
>>>> William
>>>>
>>>>
>>>>> Ray
>>>>>
>>>>>
>>>>> On 6/13/2015 11:28 PM, William Blevins wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Jun 13, 2015 at 5:14 PM, Ray Sheppard <rsheppar at iu.edu> wrote:
>>>>>
>>>>>> Hi All,
>>>>>> For what it is worth, without this functionality, use of SCons on
>>>>>> supercomputers is problematic.
>>>>>> Ray
>>>>>>
>>>>>> On 6/13/2015 10:44 AM, Vasily wrote:
>>>>>>
>>>>>>>
>>>>>>> Yes, I was talking about those. I believe you can override their
>>>>>>> default values by passing them as arguments, like
>>>>>>> $ scons all INSTALL_ROOT=some-path
>>>>>>>
>>>>>>> That would need some testing, though - I did not try setting it to a
>>>>>>> directory *not* under SConstruct's directory.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Vasily
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>> It's not clear what you are implying.
>>>>>
>>>>> SCons supports setting variables on the command line, and SCons
>>>>> supports installing outside the SConstruct directory tree, and
>>>>> supercomputing has nothing to do with either of these items.
>>>>>
>>>>> V/R,
>>>>> William
>>>>>
>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing listScons-users at scons.orghttps://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing listScons-users at scons.orghttps://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>>
>> _______________________________________________
>> Scons-users mailing listScons-users at scons.orghttps://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150614/09645534/attachment-0001.html>
More information about the Scons-users
mailing list