Shed Skin 0.1.1

I have just released version 0.1.1 of Shed Skin, an experimental (restricted) Python-to-C++ compiler. It comes with 5 new example programs (for a total of 35 example programs, at over 10,000 lines) and several important improvements/bug fixes. See here for the full changelog.

The most interesting new example is minilight, an elegant raytracer (more precisely, a global illumination renderer) that utilizes triangle primitives and an octree spatial index. As shown on the minilight homepage, it becomes up to 100 times faster.

Other new examples are Peter Norvig's sudoku solver (which unfortunately doesn't become faster, but is a cool example anyway), yet another raytracer and a mastermind strategy evaluator by Raymond Hettinger.

The fifth new example, an interactive circle packing program, is especially nice, because it is, well, interactive, but also because it shows how easy it is to generate an extension module with Shed Skin and use this in a larger program (a Pygame application in this case).

The biggest improvement (if you can call it this) for this release was to drop support for generic types. This simply means that functions that can be called with different kinds of types of objects for a single argument are no longer supported (unless these types have a common base-class, of course). Similarly, generic datastructures are no longer supported. Dropping support for generic types made the compiler core a lot simpler (and 10% smaller!), and removed the need for the -i command-line option.

Seeing the compiler core shrink so much, I was inspired to further refactor several messy areas of the compiler for readability (most notably in infer.py). This means the compiler core has also become a lot more readable/hackable with this release. I hope to continue this work of refactoring (and adding docstrings) with each release from now on, and invite anyone to help out here.

In general, I would really like to receive more help. For example, if someone could take over maintainership of the Windows version (and possible upgrade the MingW distribution that is packaged with it), that would be great. A suggestion on the coding side might be to add keyword support to generated extension modules (see extmod.py), or to optimize some builtins (for example, string/list slicing, see lib/builtin.?pp). But I would also be happy to receive more interesting test cases (most come from a single person at the moment) and/or more bug reports (which I get far too few of.)

Minilight Compiled

(Shed Skin is an experimental (restricted-)Python-to-C++ compiler.)

Minilight is an elegant minimal global illumination renderer, or raytracer, that uses triangle primitives and an octree spatial index. The original version consists of about 1,000 lines of C++, but there are several translations available at the homepage, such as ~500 line OCaml and Python versions. The Python version apparently clocks in 173 times later than the C++ version, while the OCaml version is only about 3 times slower.. Wouldn't it be interesting if we could make the Python version run faster than the OCaml version?

After some small changes (splitting up some dynamic variables, replacing calls to 'map', 'type'..), and applying some minor fixes to Shedskin, I was able to compile the Python version to C++ using Shedskin SVN. For the basic Cornell box example, it runs about 35 times faster than the modified Python version (which should again be a bit faster than the original Python version). It's probably still somewhat slower than the OCaml version, but not that much. To try to compile it yourself, you'll probably want to have more than 512 MB of RAM, and use the 'shedskin -i' option. The modified version can be found in /examples/minilight in SVN.

I'm guessing that the Python version can be made a bit faster in itself. What typically happens then is that the C++ version will get relatively even faster, because certain bottlenecks that don't really slow down Python will slow down C++ (such as allocating lots of objects). If you'd like to try and help beat OCaml, I'd be very interested in hearing about potential improvements (where readability doesn't suffer too much, obviously.)

Another program that I managed to compile, while much smaller, is one that solves a well-known chess puzzle. To compile knight2.py, I had to replace the 'key' argument to 'sorted' with a 'cmp' version (as 'key' is not yet supported), and fix a small bug in Shedskin ("'%*d' % (2, 7)" and such did not work yet.)

I'm always interested in hearing about other interesting test cases for Shedskin, as they seem hard to come by. Please also consider sending in bug reports. Both types of feedback are essential for me to keep working on Shedskin!

Update: After some tweaking, it is now more than 60 times faster than the original here. See the comments for more information.

Update 2: I removed the -i option with Shedskin 0.1.1, so it is not necessary anymore.

Shed Skin 0.1

(Shed Skin is an experimental (restricted-)Python-to-C++ compiler.)

After several years of development and thirty releases, I have finally released Shed Skin 0.1. While still experimental, all the basic pieces are now in place:

-type inference that works for many smallish programs
-support for a useful subset of Python
-support for a good subset of the standard libraries (about 17 modules, such as random, re, os, os.path..)
-extension module generation (including extension classes)
-support for the most common platforms

For this release, I was able to add a Jpeg decoder, at 1,200 lines, an iPod Shuffle programmer, at 600 lines, and a mastermind program to the set of example programs. This brings the total number of example programs to 30, at about 9,000 lines in total.

For Shed Skin 0.2, I hope to be able to focus on improving type inference scalability. So far, I've spent surprisingly little time on type inference itself, because what I had worked well enough for most test cases. I have quite a few ideas to further improve scalability, and am sure some of these will help a lot.

This release would not have been possible without the generous help of Google (through their GSOC and GHOP programs) and the following people (please let me know if I missed someone):

-B********e
-Brian Blais
-Paul Boddie
-Djamel Cherif
-Mark Dewing
-James Coughlan
-Michael Elkins
-FFAO
-Luis M. Gonzales
-Karel Heyse
-Denis de Leeuw Duarte
-Van Lindberg
-David Marek
-Jeff Miller
-Joaquin Abian Monux
-Harri Pasanen
-SirNotAppearingInThisTutorial
-Dave Tweed
-Jaroslaw Tworek
-Pavel Vinogradov

Finally, I dedicate this release to Hai Fang Ni. Thanks for all your support, and a happy Chinese newyear!!

Some nice additions to /examples

(Shed Skin is an experimental (restricted-)Python-to-C++ compiler.)

Because the set of example programs available at the Shed Skin homepage (ssprogs.tgz) hasn't changed in a while, I decided to add some nice programs to it. With some relatively simple (but as usual, important) fixes, I was able to add support for a Jpeg decompressor (!), at 1200 lines, an iPod shuffle programmer, at 600 lines, and a multi-module mastermind program, at 300 lines.

I will be adding these programs to ssprogs.tgz with 0.0.31 (or 0.1), but for now they can be found here. They should compile using Shed Skin SVN (see the note in the top of rdb.py; TonyJpegDecoder.py should be run with two arguments, e.g. "python TonyJpegDecoder.py tiger1.jpg tiger1.bmp").

Thanks to ********** for sending me the jpeg decompressor, and the authors of these programs for the nice test cases!

I would have probably added more programs (and fixed more things) if I could find more interesting test cases, but these seem hard to come by. If you happen to know of any nice programs between 500 and, say, 2,000 lines that might be statically compiled, possibly after some minor changes, please let me know. Some more bug reports would also be very welcome.

Shed Skin 0.0.30

I have just released version 0.0.30 of Shed Skin, an experimental (restricted) Python-to-C++ compiler.

Most importantly, this release adds (efficient) support for user-defined classes in generated extension modules, which should make it much easier to integrate compiled code within larger projects. More specifically, compiled classes can now be instantiated on the CPython side, and instances can be passed freely between CPython and Shed Skin without any conversion taking place. (Instances of builtin classes are still (recursively) copied, though, at the moment..)

Another major improvement was contributed by FFAO: a new 'set' implementation, directly based on the CPython code. While I haven't tested it on many benchmarks, it is clear that is now much faster, and on one benchmark it even outperforms CPython on my system by about 35%.

Other notable changes include complex number support, mapping None to NULL instead of 0 and printing it as 'None', as well as an important type inference fix.

With support for user-defined classes in extension modules, it looks like all the major pieces are now there to do a 0.1 release. The only thing I'd really like to do before that, is to improve support for the 'os' module. Please let me know if you'd like to help out here! Hopefully, with many details out of the way, I can have another good look at type inference for 0.2..

Shed Skin in the News

(Shed Skin is an experimental (restricted-)Python-to-C++ compiler).

Because a new version (0.0.30) is still some weeks away, I decided to post, in the meantime, some links to other people's blog entries about Shed Skin.

making-python-math-196-faster-with-shedskin
taking-on-shed-skin
more performance python
compiled kamaelia

Please let me know if I have missed an interesting entry.

And if you are really bored, here are also some links to (old) discussions on slashdot and osnews:

slashdot article
osnews article

Shed Skin 0.0.29

I have just released version 0.0.29 of Shed Skin, an experimental (restricted) Python-to-C++ compiler. It's been a while since the last release (well, just under 4 months), because of work and vacation, but mostly because there have been so many changes.

Thanks to the work of Karel Heyse, Pavel Vinogradov, FFAO and David Marek, there is now a pretty much complete implementation of the datetime module. Thanks to a suggestion by Albert Hofkamp, I added support for most of the ConfigParser module (by compiling it with Shed Skin of course), which led me to fix several outstanding but important problems. For example, support for inheritance hierarchies was greatly improved, and mapping keys should now work, too ('%(key)x..' % some_dict).

Since the previous release, I've also gone through the pains of installing FreeBSD, OpenSolaris and 64-bit Ubuntu on a spare PC and testing Shed Skin on each of them, with the result that these platforms are now 'officially' supported. Please see the updated 'installation' section of the tutorial. I also received some GCC 4.3 patches (Seo Sanghyeon and Winterknight), which should work fine now, too. So I guess most platforms should be covered now.

Some other improvements:

-improved support for importing from nested modules such as os.path
-__init__ methods are much less a special-case
-improved support for calling ancestor methods (e.g. Parent.__init__)
-all example programs (ss-progs) now compile as extension modules
-staticmethod and property decorator support (Seo Sanghyeon)
-Shed Skin doesn't crash on highly dynamic/recursive types anymore, making it easier to debug programs to get them to compile
-several fixes in the re module, e.g. re.sub now accepts a replacement function
-tuple hash caching was disabled, as CPython doesn't do this either

For the full changelog, see the release notes wiki at the Googlecode site. Please try out the new release, and let me know about any problems. Note that I probably won't be very responsive for at least a week or so - it has been a somewhat difficult release, and I could use some rest.. :)