Saturday, September 10, 2011

Shed Skin 0.9

I have just released version 0.9 of Shed Skin, a restricted-Python to C++ compiler. Looking at the state of things, a 1.0 version is probably not far off. There will probably be several 0.9.x releases first, though.

There were two issues blocking a 1.0 release in my mind, both of which seem to have been fixed with this release.

First, I was not completely happy yet with the type inference scalability. With 0.9, there has been another dramatic improvement in this regard. For example, the C64 emulator is analyzed 10 times faster now (about 3,000 lines in 2 minutes on my PC). But most other example programs are now analyzed much faster. It now seems a realistic option to compile a 10,000 line program.

Second, there were some long standing nasty issues with combining 'incompatible' types (such as for '[[1.0]] == [[1]]', we are comparing different types). Shed Skin 0.9 includes a new framework for dealing with these issues, and the problem seems mostly solved at this point.

There were also some very nice performance optimizations for this release. Francois Boutines greatly optimized file I/O. Complex numbers are now copy-by-value, hence massively faster. And the idiomatic construct 'for .., .. in somedict.iteritems()' should be a lot faster as well now. Thanks to Andreas van Cranenburgh for triggering the latter (as well as some of the type inference improvements) with a nice new example program, a natural language parser.

Shed Skin 0.9 also supports three new modules. Francois Boutines added support for the 'mmap' module. Fahrzin Hemmati added support for 'binascii'. And Tony Veijalainen added support for 'colorsys'.

Further, there has been a massive amount of fixes. Thanks to Jason Ye for triggering several of these. Brent Pedersen fixed some 'set' issues. Francois Boutines fixed a string comparison bug involving \0, and improved 'universal mode' for files. Joris van Zwieten sent in the magic code to only print tracebacks (shedskin -x) for uncaught exceptions.

As usual, more details can be found in the release notes.

In the meantime, Paul Boddie has been at work trying to get Shed Skin 0.9 accepted as an official Debian package. It is already included with Fedora (thanks to Thomas Spura) and some other distributions, but I'm a Debian/Ubuntu user myself, so I would be really pleased to see this happen.

Sunday, June 19, 2011

Shed Skin 0.8, Programming Language Benchmark

I'm happy to announce the release of Shed Skin 0.8, a (restricted-)Python-to-C++ compiler. It must be one of the best releases so far, with help and feedback from many directions. I'm especially thrilled to see an updated version of the C64 emulator example run one of my all-time favorite games.. :-) The emulator is now well over 3,000 lines (sloccount), and still compiling fine every time. The analysis time is in line with the scalability graph I published earlier. The game goes from a few FPS using CPython to a fluent 50 FPS after compilation on my PC.

The emulator triggered me to finally add support for the 'struct' and 'array' modules to Shed Skin. The struct module is now used to load programs from tape, and the array module is used to maintain a packed version of the screen data.

Another interesting new feature, also triggered by the emulator, allows generated code to print tracebacks for every raised exception (shedskin -x). So it's not entirely the same as in CPython (yet), which only does this for uncaught exceptions, but it's a step in the right direction. Unfortunately, I discovered at the last moment this doesn't work in the windows version. Hopefully this can be fixed for 0.9.

More details about the release can be found in the release notes.

Before I could make this announcement, there was a small performance comparison published, comparing different languages and implementations (such as PyPy, Jython and Shed Skin 0.8) for 4 different benchmarks. Unfortunately, Shed Skin 0.8 fails on one, but the problem is already fixed in GIT (regular expression search on empty string). The performance on the dict test can probably be attributed to the currently still slow I/O in Shed Skin. Improving this will be one of the goals for 0.9. Hopefully by the time you read this the comparison will have been updated at least for the failing test.

update: The file I/O bottleneck was fixed (thanks francois), and the performance comparison has been updated. Note that the first two benchmarks can still run a lot faster when disabling index-out-of-bounds checking (shedskin -b). On my PC, they become 30% and 45% faster, respectively..

update 2: the glitches are now also gone from the C64 example in GIT.

Sunday, February 20, 2011

Shed Skin 0.7.1, Help needed!

I have just uploaded the tarball for Shed Skin 0.7.1, which fixes several important problems in 0.7, and adds a few minor features. The Windows version and hopefully the Debian/Fedora packages will follow shortly. As usual, please see the release notes for an overview of all the changes.

There aren't any major new features, because I haven't received much feedback since 0.7. Please let me know if you run into any problem, so it can be fixed, or if you know of some interesting program I could play with! I'd also like to take this opportunity to ask for some help in specific areas.

Excitingly, Thomas Spura was able to get Shed Skin 0.7 accepted in the Fedora repositories! :D Thanks to Thomas for that! Inspired by this, I'd like to hear from anyone who might be interested in assisting/motivating Paul Boddie to get his Debian package accepted in the Debian/Ubuntu repositories. I think if it got past the scrutiny of the Fedora people, this shouldn't be too hard.. :-)

I would also really appreciate to see some description of Shed Skin on Wikipedia (again), because I think it deserves this, especially since the scalability improvements in 0.6. But also of course because it sends lots of users to the homepage. If you'd like to work on this, please see this thread with links to independent blog entries and even a book that describe or benchmark Shed Skin. For a summary of how the type inference part works, please see my recent 0.7 announcement.

Since a while, inspired by the LibreOffice project, there has also been a page in the Shed Skin wiki that lists some easy tasks that one could try and tackle, to hopefully make it easier to start contributing to Shed Skin. I try hard not to fix these problems myself, in the hope someone else will pick them up.

update: I just added another great new example to the 0.7.1 tarball, a quantum monte carlo simulator by Mark Dewing. shedskin takes about 10 minutes on my PC to analyze its 1,200 lines of code (sloccount).