[Scons-users] Doing a build time Glob()

[Pico Geyer]

Hi Marc,


On Thu, Nov 28, 2013 at 6:05 PM, Marc Branchaud <marcnarc at xiplink.com>wrote:


> On 13-11-28 04:04 AM, Pico Geyer wrote:

> > Hi all.

> >

> > I'm trying to build an external project (Python) by calling

> configure/make

> > from scons and then Install() a subset of that projects files into my own

> > directory.

> >

> > I'm not having trouble with build part of the project, however the

> Install

> > part goes wrong.

> > I naively started with something like this:

> > env.Install('mydir/python/lib/python2.7',

> > Glob('python_install_dir/lib/python2.7/*.py'))

> >

> > This of course doesn't work because the Glob() gets run at the SConscript

> > reading and at that stage the desired files don't exist yet (My

> dependencies

> > are setup correctly to build python before trying to install the files)

> > If I run my install target twice however, the files will exist and then

> be

> > installed correctly.

> > It's be very clumsy to tell the users of the build to run the install

> target

> > twice, so I need a better solution.

> >

> > I don't suppose there's any built in way to run the Glob at build time,

> when

> > the files will exist?

>

> This is one of my top frustrations when using SCons to run external

> (non-SCons) builds.  AFAIK there is no built-in way to do what you ask.

>

> SCons desperately needs to know about every file it deals with:  Either the

> file is an original source (that exists in the "clean" project) or SCons

> knows how to build the file.

>

> I too have wrangled SCons into building and installing Python.  My case is

> a

> bit different from yours, in that I only install a subset of Python's

> modules.  My SConscript file literally lists *each*and*every* file I care

> about that Python's "make install DESTDIR=..." creates (including the

> Python

> binaries and other files).  This list forms the targets of the Builder that

> runs the make command.  I then Install() those as I like.

>


It seems our use cases are similar. I want a minimal python install. So
using all the python files in the Lib directory is more a first attempt.
I share your frustration because it seems very time consuming and fragile
to list every file manually in your Install().
Luckily a particular python release does not update it's modules too
frequently (right?).
But this becomes even more troublesome when you want the latest version of
some project that might update it's files frequently.
Also imagine a project like boost that has > 10 000 header files.


>

> I would dearly like for SCons to allow me to say something like "Just copy

> whatever files you happen to find in Directory A to Directory B".  But I

> think that fundamentally this would require SCons to support dynamic (i.e.

> build-time) modifications of its dependency tree.

>

> In your case, if you don't want to list all the files, I see a couple of

> options:

>

> 1. Use AddPostAction() to do a "cp * foo/" after you run the Python build.

>

> 2. Write a custom MyInstall() builder that runs "cp * foo/".

>

> Both of those options can be problematic if you want to have something else

> depend on what you install.  Even implicit dependencies will get tripped

> up:

> Suppose after building Python you want to build some C code that uses some

> of

> the Python include files.  If you do this, the first time you build Python

> and your C code it'll work fine.  But if you then do an incremental rebuild

> of your C code SCons will delete the Python .h files your C code uses!

>  This

> is because you haven't told SCons how those include files came to be, and

> when SCons sees that your C sources depend on those files it'll delete them

> as it's scanning dependencies.

>


I think the above is only a problem if you build python (or external
project) into your variant dir.

I've started following the policy of not installing any third party
projects into my variant dir.
This seems to side step the problem that you mentioned.




> (When building an external library with SCons, my main Builder that runs

> "make install" in some way has as its targets the .a and/or .so files, and

> I'll use a SideEffect() of that builder to tell SCons about the generated

> .h

> files.)

>

> So if you need something to depend on the Python stuff you install, you

> have

> no choice but to list that stuff explicitly.

>

> (With Python's library of .py files, the installed ones are basically

> copied

> from the Lib/ directory in Python's source tree, so you could use a Glob()

> of

> the *.py files there as the basis of a list of the installed *.py files.

> However, the Python build also creates a bunch of .so files it installs

> under

> lib/python2.7/lib-dynload/, and you need some of those to have your

> Install()'d Python work at all.  I think these .so are kinda-sorta derived

> from the .c files in the Modules/ directory, but I dunno for sure.)

>


Ah, using the original Lib dir is a good idea.
So now the problem is down to just the generated .so (or pyd) files
To make things worse, I need my python installation to work on Windows,
where things are a bit different.



>

> After struggling with this sort of problem for a long time, I've given up

> trying to be clever and I always just list everything an external build

> command creates.  It's a hassle to maintain the SConscripts as the external

> code changes, but with SCons it's the best solution I've found.

>


I might still be too naive but I'm going to try get things working without
listing every python file in my Sconscript file.

Thanks for your response!
Regards,
Pico
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://four.pairlist.net/pipermail/scons-users/attachments/20131129/9a36bc87/attachment-0001.htm