Readline and Python 2.5+ on OS X

This is really just a “note to self” kind of post. I meant to write this down a while ago but I forgot. To prevent further forgetting, here it is:

I always compile Python myself. The Python that comes with OS X tends to get outdated pretty soon and it has outdated libraries in its site-packages directory. And beware installing or updating anything in there, it might ruin core components of OS X (because they actually use this Python instead of one that’s not the user’s to modify). I know that MacPorts too has various Python versions but it then again it applies various patches to them and builds them in weird manners that I don’t understand (framework, etc.). So the best bet to get a clean and reliable Python installation is to self-compile (and then use virtualenv to prevent it from being messed up).

As it happens, when I compiled Python 2.5 or higher on OS X, it linked to either the OS X readline library or the MacPorts one. Which one I don’t know, but it was definitely hosed. So while the interpreter worked fine, the interpreter shell would crash with a Bus Error. So what I did was compile my own plain vanilla version of readline and installed it to /opt. Of course that didn’t work right away because readline wouldn’t build on OS X Leopard without applying a small patch to a build script.

After having installed readline, I configured Python with the (undocumented, but apparently existing) --with-readline-dir option:

./configure --prefix=/opt --with-readline-dir=/opt

and did the usual make && make install dance.

Chameleon: byte compiler for ZPT and Genshi

If you’re a Python web developer and are interested in fast templating, Chameleon might be of interest to you. Brought to you by that crazy Austro-Danish duo Malthe Borch and Daniel Nouri, it’s a byte compiler that bakes HTML/XML templates into Python (byte) code. Currently it has support for Zope Page Templates and Genshi. And it’s fast. Fricken fast. If I remember correctly, it doesn’t achive the same speeds as Google’s Spitfire in all benchmarks, but it’s in the same league.

Now Malthe and Daniel as well as some regular contributors such as Chris McDonough, Wichert Akkerman and Hanno Schlichting are perfecting ZPT and Genshi compatibility. With support for macros and i18n, it already looks like a serious contender to replace zope.pagetemplate in templating-heavy Zope apps such as Plone. In fact, Chameleon might be the perfect match for Alex Limi’s proclaimed faster and lighter Plone 4, next to ditching Archetypes, reducing the Component Architecture overdose and going more Pythonic.

That said, it’d be very interesting to see Chameleon being tried by the non-Zope crowd. repoze.bfg has already adopted it as its de-facto standard templating engine. Has anyone tried it with Django, Pylons, TurboGears yet?

Cheap binary Windows eggs

(This is a post from my old blog which seems to be going offline once in a while, maybe even permanently at some point. This article is still very useful so I’ve reposted it here for other people’s and my reference.)

Thanks to Hanno Schlichting’s howto, I’ve figured out how to create Windows eggs of those packages that have C extensions. This approach doesn’t need Microsoft Visual Studio, nor does it require you to wade
through a bunch of free Microsoft downloads that don’t really work in the end anyway.

Here’s what I did:

1. Installed the standard Python 2.4 distribution from the MSI package.
2. Installed the MingW compiler (into the standard location C:\MingW)
3. Created C:\Documents and Settings\Philipp\pydistutils.cfg and put the following text in it:
   

   [build]
   compiler=mingw32

   This tells distutils to always use the MingW compiler whenever it has to compile something from C.

4. Went to the Control Panel -> System -> Advanced tab and clicked on the Environment Variables button. There I appended the following text to the the Path environment variable, adding the Python interpreter as well as MingW’s programs to the search path:
   

   ;C:\Python24;C:\MingW\bin

   Then I added another environment variabled called HOME with the following value:

   C:\Documents and Settings\Philipp

   This points distutils at the pydistutils.cfg file that I created earlier (you can put the pydistutils.cfg file anywhere you want, you just need to make sure that the HOME environment
   variable points to the directory).

5. With that in place, I am able to take any tarball (e.g. zope.interface-3.4.0.tgz), unzip it and create a Windows egg from it like so:
   

   python setup.py bdist_egg

What’s more, with a setup like this, it is easily possible to install Zope 3 completely from eggs (e.g. using zc.buildout) even if there are no pre-built Windows eggs on the Cheeseshop. More specifically, with this setup, zopeproject (which is really just a convenience tool over zc.buildout) works like a charm on Windows now.

Version numbers on trunk and maintenance branches

The release process that I wrote for the Zope subversion repository states that a library’s version number on the trunk or a release branch should always be the next release version number applicable to that branch. For instance, if zope.interface 3.4.1 were just released from the 3.4.x branch, the version number of zope.interface on that branch should read 3.4.2dev.

Let me explain why I suggested this practice and, despite much critique, still maintain it’s makes the most sense.

First of all, the setuptools documentation states:

Note: the project version number you specify in setup.py should always be the next version of your software, not the last released version.

So it’s a convention that seems to be generally suggested. That doesn’t necessarily mean it’s a good idea, though.

What makes it a good idea is the fact that when you get a checkout of the trunk or a development branch, the version number is actually meaningful, due to setuptools’ version semantics

3.4.1 < 3.4.2dev < 3.4.2

So, a development egg of zope.interface 3.4.2dev will for instance satisfy a version requirement like “zope.interface > 3.4.1“. For example, say you wanted to temporarily deploy from a subversion checkout. I had to do this to get my PyFlakes running with Emacs. The latest release PyFlakes 0.2.1 wasn’t good enough because something got fixed on the trunk. So I got a trunk checkout and did a

python setup.py install

into a virtualenv. Surely enough, the trunk’s version number was still pointing to 0.2.1. So I ended up with pyflakes-0.2.1-py2.4.egg in my site-packages and no way to tell it from the actual 0.2.1 release. (Yes, I know there’s some setuptools parameter than can let you build versions like 0.2.1-r48292, and those would be fine, if they were configured to occur on the trunk automatically.) So that’s why not only bumping the version number to the next release but also adding the “dev” marker to tell development eggs or snapshots apart from actual release (and prevent people from releasing from the trunk!) is a good idea.

To conclude, I think bumping the version number to the next or at least the next anticipated release (it’s ok if you don’t get the version number right the first time) and adding a dev marker is not only a good idea, I think it’s pretty much the only way to get the version number semantics of development eggs right (unless you use r34234 suffixes which have other problems). Of course, I’m willing to be convinced otherwise if alternate solutions achieve the same semantics. Comment away!

Is there a point to distributing .egg files?

So far I’ve found that distributing .egg files is mostly useless:

• Source tarballs as created with the distutils/setuptools sdist command are not only equivalent to eggs, they often contain more information (such as top-level README.txt or INSTALL.txt files). Also, not everybody has embraced easy_install yet and a tarball is the least surprise to the old-school folks.
• .egg files are marked with the Python version they were created with. So if you only upload an .egg file that was created with, say, Python 2.4, and you don’t provide a source tarball, Python 2.5 users will be out of luck trying to easy_install your package (even though it may perfectly work on Python 2.5).
• If the package contains C extensions, you pretty much can’t risk uploading an .egg file because it’ll contain binaries. With Linux and MacOSX this is unacceptable due to the various ways Python itself can be built on these platforms (and will therefore likely be incompatible with anything you’ve built). One notable exception is Windows which (thanks to its homogeneousness) makes it possible to distribute binaries w/o problems. Then again, because Windows users rarely have the right compiler installed, it pretty much requires you to distribute binaries.

So why are we still uploading .egg files to PyPI? Isn’t it enough and even better to just upload the source tarball? (And a Windows binary egg only if the package contains C extensions.)

setup.cfg considered harmful

ZopeSkel is a neat collection of Zope-related PasteScript templates, but I have two beefs with it. First of all, most templates ask way too many questions (why does it want a description of my package as reST now???), but I can deal with that (just hit return a bunch of times until the annoyance is over). What’s very annoying is the setup.cfg file it generates. It needs to go.

Here’s why: It sets a bad example. The setup.cfg files that ZopeSkel generates are configured so that they adds the ‘dev’ marker and the current SVN revision number to the egg’s version number. In other words, ZopeSkel suggests that continuous releases are a good idea. They’re not. If you don’t want to deploy from a release, then just get a checkout and deploy a development egg. If you want to deploy from a release, make a release (yes, I know that it takes time, but we have release procedures for a reason). Of course, when you want to make a proper release and you forget about setup.cfg, you will generate a fubared release (e.g. foo-1.3dev-r2342 instead of foo-1.3).

It does make sense to mark the trunk or development branches with the ‘dev’ marker (so that accidental releases don’t happen), but the logical places for this is setup.py (because that’s where you’d expect to set the version number for a release).