Comments on: Is there a point to distributing .egg files? http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/ Philipp on software and other interesting things Thu, 01 Oct 2009 18:05:34 +0000 http://wordpress.com/ hourly 1 By: Chris Galvan http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-279 Chris Galvan Sat, 27 Dec 2008 06:18:30 +0000 http://philikon.wordpress.com/?p=11#comment-279 @Ryan: I'm not sure exactly what problem you are explaining :) Using setuptools, you can have the same namespaces in different eggs and they won't overlap(overrule) eachother, if this is what you are afraid of. Instead, the namespace would be extended to contain the contents from each egg. Was this the problem you were describing or am I way off? :) @Ryan: I’m not sure exactly what problem you are explaining :) Using setuptools, you can have the same namespaces in different eggs and they won’t overlap(overrule) eachother, if this is what you are afraid of. Instead, the namespace would be extended to contain the contents from each egg.

Was this the problem you were describing or am I way off? :)

]]> By: Justin Ryan http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-161 Justin Ryan Sun, 21 Sep 2008 21:07:56 +0000 http://philikon.wordpress.com/?p=11#comment-161 Whups, there should have been a comma after __init__.py above, and a lower 's' on surely. Whups, there should have been a comma after __init__.py above, and a lower ’s’ on surely.

]]> By: Justin Ryan http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-160 Justin Ryan Sun, 21 Sep 2008 21:06:57 +0000 http://philikon.wordpress.com/?p=11#comment-160 Has anyone considered that the whole idea of eggs as deployment is a bit of a mess as well? I haven't profiled, but a lot more work must be happening to traverse more numerous, deep directory structures, many of whom are mirrors of each other, confusing the always-beautiful packages-are-directories meme that makes Python so easy to deal with at a large scale, or should. When you have a package like plone.* or zope.* which has 5, 10, 20 subpackages depending on how many levels you look at, and each of those has its' own entire tree with a 'zope' or 'plone' subdirectory and some boilerplate __init__.py. Surely there must be some overhead to reconciling these mirrored or, depending on how you look at it, overlapping namespaces. I agree that, eggs or sdist, it's nice to have a simple, atomic package representing a given bit of functionality which may work on its' own or at least without *all* of the packages from the master, but it seems that omelettes are ideal for deployment, once you've selected the packages. What we do now in a typical buildout is akin to extracting each package for a gnu/linux system, and symlinking its' contents into place, and symlinking the contents into another place. Who's with me!? ;) Has anyone considered that the whole idea of eggs as deployment is a bit of a mess as well? I haven’t profiled, but a lot more work must be happening to traverse more numerous, deep directory structures, many of whom are mirrors of each other, confusing the always-beautiful packages-are-directories meme that makes Python so easy to deal with at a large scale, or should.

When you have a package like plone.* or zope.* which has 5, 10, 20 subpackages depending on how many levels you look at, and each of those has its’ own entire tree with a ‘zope’ or ‘plone’ subdirectory and some boilerplate __init__.py. Surely there must be some overhead to reconciling these mirrored or, depending on how you look at it, overlapping namespaces.

I agree that, eggs or sdist, it’s nice to have a simple, atomic package representing a given bit of functionality which may work on its’ own or at least without *all* of the packages from the master, but it seems that omelettes are ideal for deployment, once you’ve selected the packages.

What we do now in a typical buildout is akin to extracting each package for a gnu/linux system, and symlinking its’ contents into place, and symlinking the contents into another place.

Who’s with me!? ;)

]]> By: Chris Galvan http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-49 Chris Galvan Fri, 27 Jun 2008 15:22:19 +0000 http://philikon.wordpress.com/?p=11#comment-49 @philikon: Yes, that is a more difficult issue to solve and to account for specific compile-time settings in the egg identifiers could clutter the egg names pretty quickly :) In order to circumvent these issues, we (Enthought) ship a fully built version of Python with our extra packages included that can be installed along-side existing Python installations without conflicts. You can then get updates to these packages from our egg repositories, which have all been built against a Python installation that will be compatible with the end-users'. While this doesn't solve the issue you mentioned for user's who simply want to install a few select packages within their current installation, it does solve our large-scale distribution problem. @philikon: Yes, that is a more difficult issue to solve and to account for specific compile-time settings in the egg identifiers could clutter the egg names pretty quickly :)

In order to circumvent these issues, we (Enthought) ship a fully built version of Python with our extra packages included that can be installed along-side existing Python installations without conflicts. You can then get updates to these packages from our egg repositories, which have all been built against a Python installation that will be compatible with the end-users’.

While this doesn’t solve the issue you mentioned for user’s who simply want to install a few select packages within their current installation, it does solve our large-scale distribution problem.

]]> By: philikon http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-48 philikon Fri, 27 Jun 2008 14:52:42 +0000 http://philikon.wordpress.com/?p=11#comment-48 @Chris: That's interesting, though my experience with Linux binaries has been far more problematic just rpath problems. For instance, some Linux distributions compile their Python with the UCS4 setting for unicode, some with UCS2 or whatever it is. As far as I can tell, there's no way to work around that and the egg identifier doesn't account for this compile-time setting. @Chris: That’s interesting, though my experience with Linux binaries has been far more problematic just rpath problems. For instance, some Linux distributions compile their Python with the UCS4 setting for unicode, some with UCS2 or whatever it is. As far as I can tell, there’s no way to work around that and the egg identifier doesn’t account for this compile-time setting.

]]> By: Chris Galvan http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-47 Chris Galvan Fri, 27 Jun 2008 14:35:11 +0000 http://philikon.wordpress.com/?p=11#comment-47 Without any additional modifications, yes eggs do not lend themselves as a distribution mechanism, but there is a potential there that has not been realized by most people. The main obstacle is binary compatibility, but this can be overcome with post-install scripts that fix up the rpaths and mach-o headers, for Linux flavors and Mac OS X respectively, of the binaries inside the eggs. As an example, hdf5's extensions are linked against zlib libraries and you can't be certain what version of zlib an end-user may have on their system. However, if you ship the zlib runtime libraries with the egg and then fix up the rpaths(or mach-o headers) to point inside the egg, you have a self-contained egg that will work regardless of the end-user's installed version of zlib. The company I work for (Enthought) uses eggs on a large-scale of distribution and so far it has worked on Windows, 2 flavors of Linux and Mac OS X. There are already tools in place that can be used to modify rpath's and mach-o headers in order to deliver dynamically-linked binaries. For Linux, there is a tool called chrpath and for OS X there is macholib. There is definitely more potential in eggs than is currently credited towards them, as I have seen them successfully used as a distribution mechanism across a range of platforms. Some changes would need to be made in setuptools though in order for everyone to take advantage of them. Without any additional modifications, yes eggs do not lend themselves as a distribution mechanism, but there is a potential there that has not been realized by most people. The main obstacle is binary compatibility, but this can be overcome with post-install scripts that fix up the rpaths and mach-o headers, for Linux flavors and Mac OS X respectively, of the binaries inside the eggs.

As an example, hdf5’s extensions are linked against zlib libraries and you can’t be certain what version of zlib an end-user may have on their system. However, if you ship the zlib runtime libraries with the egg and then fix up the rpaths(or mach-o headers) to point inside the egg, you have a self-contained egg that will work regardless of the end-user’s installed version of zlib.

The company I work for (Enthought) uses eggs on a large-scale of distribution and so far it has worked on Windows, 2 flavors of Linux and Mac OS X. There are already tools in place that can be used to modify rpath’s and mach-o headers in order to deliver dynamically-linked binaries. For Linux, there is a tool called chrpath and for OS X there is macholib.

There is definitely more potential in eggs than is currently credited towards them, as I have seen them successfully used as a distribution mechanism across a range of platforms. Some changes would need to be made in setuptools though in order for everyone to take advantage of them.

]]> By: Martin Aspeli http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-46 Martin Aspeli Fri, 27 Jun 2008 10:54:24 +0000 http://philikon.wordpress.com/?p=11#comment-46 I hadn't appreciated this before and am guilty of uploading many a .egg file. I'll stop, though. :) To PJE's point, JAR files are also the main distribution mechanism in Java land, so I think the analogy there could cause some confusion. WAR and EAR files are more of a deployment concern for web apps, but will contain many libraries and other files, like templates and configuration files, so that's not a perfect analogy either. Cheers, Martin I hadn’t appreciated this before and am guilty of uploading many a .egg file. I’ll stop, though. :)

To PJE’s point, JAR files are also the main distribution mechanism in Java land, so I think the analogy there could cause some confusion. WAR and EAR files are more of a deployment concern for web apps, but will contain many libraries and other files, like templates and configuration files, so that’s not a perfect analogy either.

Cheers,
Martin

]]> By: Brandon Craig Rhodes http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-45 Brandon Craig Rhodes Fri, 27 Jun 2008 02:30:39 +0000 http://philikon.wordpress.com/?p=11#comment-45 Many users of my PyEphem module seem not to have their "python2.4-dev" package, or whatever their distribution calls it, installed; and might not even have the right compiler installed for building extensions to Python, in addition. My impression had been that, merely by having Python on their system, it is possible for them to install "setuptools" and then "easy_install" my package — and, if I've provided a binary egg, that's all they need to do. Lacking the binary egg, they've got to figure out how to install the Python development suite for whatever operating system they are using, which is way beyond the knowledge of lots of people who just want to try writing astronomy code in this new language they're heard of called "Python". Maybe you're thinking of a population that's more developer-heavy? Many users of my PyEphem module seem not to have their “python2.4-dev” package, or whatever their distribution calls it, installed; and might not even have the right compiler installed for building extensions to Python, in addition. My impression had been that, merely by having Python on their system, it is possible for them to install “setuptools” and then “easy_install” my package — and, if I’ve provided a binary egg, that’s all they need to do. Lacking the binary egg, they’ve got to figure out how to install the Python development suite for whatever operating system they are using, which is way beyond the knowledge of lots of people who just want to try writing astronomy code in this new language they’re heard of called “Python”.

Maybe you’re thinking of a population that’s more developer-heavy?

]]> By: Brodie Rao http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-44 Brodie Rao Fri, 27 Jun 2008 02:01:42 +0000 http://philikon.wordpress.com/?p=11#comment-44 I always use sdist when uploading to PyPI, and I've personally stopped explicitly supporting setuptools in my setup.py scripts. While I love using easy_install myself, I prefer to have a more consistent user experience in terms of installation, and I was basically completely turned off to it when I discovered that it "sandboxes" setup scripts, not allowing them to write outside of the package's path/egg. I don't think it's worth the trouble to support, it doesn't provide that many compelling features on the package maintainer's end, and you can still usually just easy_install any source distribution anyway. Now what I'd really like to see is an easy_install that doesn't use setuptools and just searches and downloads from PyPI. I always use sdist when uploading to PyPI, and I’ve personally stopped explicitly supporting setuptools in my setup.py scripts. While I love using easy_install myself, I prefer to have a more consistent user experience in terms of installation, and I was basically completely turned off to it when I discovered that it “sandboxes” setup scripts, not allowing them to write outside of the package’s path/egg.

I don’t think it’s worth the trouble to support, it doesn’t provide that many compelling features on the package maintainer’s end, and you can still usually just easy_install any source distribution anyway.

Now what I’d really like to see is an easy_install that doesn’t use setuptools and just searches and downloads from PyPI.

]]> By: philikon http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-42 philikon Thu, 26 Jun 2008 19:33:58 +0000 http://philikon.wordpress.com/?p=11#comment-42 @Marius: Since regular Windows binary eggs are marked with the 'win32' platform, I imagine that those for Windows on 64bit would be marked with 'win64' or something along those lines. Also, I would expect that 'win32' eggs would work on a Windows 64bit platform. @Marius: Since regular Windows binary eggs are marked with the ‘win32′ platform, I imagine that those for Windows on 64bit would be marked with ‘win64′ or something along those lines. Also, I would expect that ‘win32′ eggs would work on a Windows 64bit platform.

]]>