[Scons-users] Scons on pypy

Sun Jun 12 01:35:40 EDT 2016

Here's a super crude SConstruct that makes a wheel file for my pysdl2-cffi
project. https://gist.github.com/dholth/b9673950b20843afc91c26982788b1e4

The trick is placing a few metadata files METADATA, WHEEL, and RECORD into
a package name- and version-specific directory, adding the regular files,
and then generating a manifest.

For wheel I want to say 'put this directory under $purelib, that directory
under $platlib, another directory under $headers', etc. and have it all map
to paths like pysdl2_cffi-1.0/headers/ , pysdl2_cffi-1.0/data/ and so on
inside the archive. It was a struggle to figure out how to do that, but I
just installed to a build directory and zipped it at the end. My conceptual
problem here is that yes, the files will be installed eventually, but not
by SCons, while Package(source=FindInstalledFiles()) assumes SCons can also
install those files.

sdist is easier than wheel, it only requires PKG-INFO and setup.py to be
recognized by PyPi as a sdist, and otherwise fits well with SCons' idea of
a source package.

All setuptools' 'develop' target does (not reimplemented in this
SConstruct) is to build the package in place, generate egg_info, and modify
a .pth file in site-packages to add the package root, usually '.' or 'src',
to sys.path.

I gather once 'pip install SCons' there would be no simple way to run it
from inside a setup.py shim, apart from subprocess?

One way to get the Python compiler arguments is to ask distutils for the
command it would use to invoke the compiler, and then have SCons
ParseFlags(). Does that function understand MSVC flags too?

Is there an Ant-style glob function that you can recommend? (featuring /**/
to match path segments, recursing into subdirectories)

How do you do substitution with $VARIABLE.not-variable ?
$(VARIABLE).not-variable or {} braces didn't seem to work.

What is the advantage of adding to CPPATH etc. with Configure() rather than
just doing it in env directly?

Thanks,

Daniel Holth

On Mon, Jun 6, 2016 at 12:52 PM Daniel Holth <dholth at gmail.com> wrote:

> "pip install -e ." is "setup.py develop". "pip install" used to invoke
> "setup.py install", but now it builds a wheel first, if that succeeds it
> installs the wheel, otherwise it falls back to "setup.py install".
>
> On Mon, Jun 6, 2016 at 12:24 PM Daniel Holth <dholth at gmail.com> wrote:
>
>> It's still allowed, just pip doesn't use it.
>>
>> On Mon, Jun 6, 2016, 11:32 Jason Kenny <dragon512 at live.com> wrote:
>>
>>> I assume you mean that we can no do a “pip install -e .” to get a
>>> developer build working, have them remove the -e to get a full install.
>>>
>>>
>>>
>>> I always tell people to use pip now if we can do it, then fallback to
>>> “setup install” when they are on some system where there seems to be an
>>> issue getting pip install easily.
>>>
>>>
>>>
>>> Do you have better advice on how that should be done?
>>>
>>> Jason
>>>
>>>
>>>
>>> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf
>>> Of *Daniel Holth
>>> *Sent:* Monday, June 6, 2016 10:03 AM
>>> *To:* SCons users mailing list <scons-users at scons.org>; dl9obn at darc.de
>>>
>>>
>>> *Subject:* Re: [Scons-users] Scons on pypy
>>>
>>>
>>>
>>> Of note is that modern versions of pip do not need 'setup.py install'.
>>> Instead, they always build a wheel first, then install that.
>>>
>>>
>>>
>>> On Mon, Jun 6, 2016 at 10:58 AM Daniel Holth <dholth at gmail.com> wrote:
>>>
>>> I don't have it yet. I would be happy to trade information about the
>>> necessary commands and Python packaging metadata for SCons advice :)
>>>
>>>
>>>
>>> List of what pip invokes on setup.py
>>> https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface
>>>
>>>
>>>
>>> Here's an implementation of setup-requires (install requirements before
>>> running the bulk of setup.py).
>>> https://bitbucket.org/dholth/setup-requires Eventually a future version
>>> of pip will do this for you, and this will become a no-op shim.
>>>
>>>
>>>
>>> Here's an experimental waf script that builds a wheel for bdist_wheel
>>> itself, in a few lines of code.
>>> https://bitbucket.org/dholth/wheel/src/tip/wscript
>>>
>>>
>>>
>>> Wheel also has code to convert between old and new metadata formats.
>>>
>>>
>>>
>>> I haven't measured the impact of __slots__ on PyPy 5.1 very much except
>>> that it prevents it from working at all. You would have to work with the
>>> PyPy developers on that front. 'local-zip' uses more time and more memory
>>> on PyPy, but there may be better SCons benchmarks.
>>>
>>>
>>>
>>> On Mon, Jun 6, 2016 at 10:18 AM Jason Kenny <dragon512 at live.com> wrote:
>>>
>>> I was hoping to clean up the setup.py file myself. I had hoped to make
>>> it use whl files. I had hoped to test it with pypy myself.
>>>
>>>
>>>
>>> I was hoping to make it simpler like the one I define for a project I am
>>> working on here:
>>> https://bitbucket.org/dragon512/reusable-gold-testing-system/src/23db072a0d7469e777f8921ca6e7580894587da4/setup.py?fileviewer=file-view-default
>>>
>>>
>>>
>>> However we seem to have the expert here J Do you have a better setup.py
>>> already, or should I still try to give it a shot and have you give it a
>>> review?
>>>
>>>
>>>
>>> Also on the __slots__ issue with pypy. I had defined a test to show the
>>> need for the node changes and posted it here some place. The use of
>>> __slots__ or __slabs__ in pypy case shows values. I did the test with
>>> ironpython, cpython 2 and 3 and pypy. All showed better use of memory. This
>>> is critical for Scons to scale better for large build. As a side effect of
>>> use less memory for the same information we get a speed boost as well, and
>>> for a build system that is good, as user always complain that the “build is
>>> slow”.
>>>
>>>
>>>
>>> Jason
>>>
>>>
>>>
>>> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf
>>> Of *Daniel Holth
>>> *Sent:* Sunday, June 5, 2016 1:00 PM
>>> *To:* dl9obn at darc.de; SCons users mailing list <scons-users at scons.org>
>>> *Subject:* Re: [Scons-users] Scons on pypy
>>>
>>>
>>>
>>> I am the guy who invented wheel (*.whl) for Python packaging, and I am
>>> interested in advancing Python packaging generally. I feel that a
>>> setuptools monoculture holds us back generally, it hails from the late 90's
>>> (distutils) and very few people understand how to extend it. SCons seems
>>> like a nice way to experiment with a more flexible build system that is
>>> written in Python and easily installed from pypi.
>>>
>>>
>>>
>>> Soon, pip will be able to install dependencies *before* it touches
>>> setup.py. See
>>> https://github.com/brettcannon/build-deps-pep/blob/master/pep-0NNN.rst.
>>> Pip is also working on defining an interface (consisting of a few setup.py
>>> commands) that it promises to use when invoking setup.py. That makes it
>>> much easier to bring abstractions to setup.py and re-implement just the
>>> parts that pip needs. So for any project that runs against the limitations
>>> of distutils there should and can be other options.
>>>
>>>
>>>
>>> As it happens I also have a concrete project that does benefit from
>>> SCons. The pysdl2-cffi binding to SDL2 is not particularly exceptional, but
>>> it needs to automatically generate some *.py code to help wrap the SDL2
>>> library before "setup.py bdist_wheel" can be called. It does this by
>>> parsing SDL2's headers with a little compiler that generates a friendlier
>>> wrapper compared to using cffi raw. It also calls cffi to generate *.c and
>>> compiles that as an extension module (here, it is especially helpful to be
>>> running under the target Python, to ask distutils for compiler options).
>>> SCons is great at this because there are several build steps that depend on
>>> each other's outputs. It would be difficult to do that efficiently with
>>> distutils or setuptools.
>>>
>>>
>>>
>>> The other aspect I'm interested in is to implement bdist_wheel in SCons.
>>> It should be much easier to prototype new features without having to deal
>>> with distutils' and setuptools' baggage.
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> Daniel Holth
>>>
>>>
>>>
>>> On Sun, Jun 5, 2016 at 12:11 PM Dirk Bächle <tshortik at gmx.de> wrote:
>>>
>>> Hi Daniel,
>>>
>>> On 05.06.2016 14:55, Daniel Holth wrote:
>>> > Simply I want to try using scons to implement setup.py. To do this I
>>> need to have scons run on whatever versions of Python the
>>> > package is compatible with, obviously. Then I need to write a task to
>>> generate a couple of packaging metadata and archive formats.
>>> >
>>>
>>> are we really talking about the setup.py from the setuptools? Do you
>>> really want to re-implement it, but using SCons?
>>> Please feel free to do so, I'm just wondering what the benefit of this
>>> would be. I mean, for installing SCons on my system I'd need
>>> to have the setuptools anyway...so what does your solution bring that
>>> setuptools doesn't have? Just curious...
>>>
>>> Best regards,
>>>
>>> Dirk
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160612/ba522472/attachment-0001.html>