Shed Skin restricted-Python-to-C++ compiler 0.9.6

I recently decided to sit down and finally port Shed Skin, an experimental restricted-Python-to-C++ compiler in the works since 2005 or so, to Python3. Three painful months and a total diff of 50k lines later, everything now works with Python3 (Shed Skin itself, and all tests and examples..)

This does not mean that every Python3 feature is supported, but what was there now at least works fine with Python3.. For example, unicode is still restricted to 1-byte characters, and there is no support (yet) for nice new features such as f-strings. Python2 support has been dropped with the new release, and subsequent releases should add support for various new Python3 features.

The following people have contributed along the way:

• Johan Kristensen (large patch for moving from compiler.ast to ast, still on Python2)
• Shakeeb Alireza (extension module support for Python3, ported many examples, improved OSX support, and various code cleanups)
• Folkert van Heusden (some fixes on the C++ side, move to c++17)
• Jeremie Roquet, Thomas Spura, Paul Boddie and others who kept doing maintenance on the project in my absence

I started work on the port after realizing that Shed Skin was being removed from distributions, especially Debian, as it was still tied to Python2, and I really want to keep Shed Skin in a working state if possible.

So what about the future of the project? Not really sure, but I'm happy to sit down again in a few months to prepare a new release. Any feedback on what to support or improve would be very welcome!

For fun, here are screenshots of some of the Shed Skin example programs (in total 75 working example programs can be found on the github site):

Shed Skin 0.9.4

I have just released Shed Skin 0.9.4, a (restricted-)Python-(2.x)-to-C++ compiler. The full release notes can be found here, as usual.

Major thanks go to Ernesto Ferro, who has been making many large refactorings in the code to improve maintainability. He also found a nice new example, called Gh0stenstein (see picture below). Paul Haeberli has also triggered several very useful improvements.

Besides refactoring, most changes are minor fixes or improved tests. But there are quite a few of them. There are also 3 new examples, meaning there are now 75 examples in total, which I think is a milestone in itself.

Shed Skin 0.9.3

I have just released version 0.9.3 of Shed Skin, a restricted-Python (2.4-2.6) to C++ compiler. This is the third maintenance release since 0.9, and as such there were (almost) no major changes.

For the full list of changes since 0.9.2, please see the release notes.

Five interesting new examples were added for this release, bringing the total to 72. Most interesting to me are Pygasus, a 1,500 line NES emulator that is able to play mario bros (Maciek "Mistrall" Żuk), as well as a pure Python implementation of timsort, the famous Python sorting algorithm (ported from java to python by Dan Stromberg).

In the meantime, the C64 emulator (see my earlier posts) has grown to about 6,000 lines of code (sloccount) but is still compiling fine (though it takes about 10 minutes here, still not that bad). I'm tempted to update the scalability graph I posted earlier.

In any case, I would like to thank the authors of the new example programs for sharing their code (maciek, dan, thomas, piotr, keegan) and those who provided improvements (danny milosavljevic) or feedback since 0.9.2.

Shed Skin 0.9.2

I have just released version 0.9.2 of Shed Skin, a restricted-Python (2.4-2.6) to C++ compiler. This is the second maintenance release since 0.9, so no new major features were added.

The biggest improvement is probably some optimizations for the array module. Other than this, the changes are mostly unrelated minor optimizations and bug fixes. Please see the release notes for more detail.

Three nice new examples were also added for this release, bringing the number of examples to 67, with a total linecount of about 17,000 lines (sloccount): a "stereo vision" program, and two Rubik's cube solvers.

Shed Skin 0.9.1

I have just released version 0.9.1 of Shed Skin, a restricted-Python (2.4-2.6) to C++ compiler. This is a maintenance release, so no new major features were added. Most interesting perhaps are optimizations for itertools.product and str.join. Other than that, it's mostly bugfixes and a new (hq2x image scaling) example. Please see the release notes for more details.

In the meantime, Paul Boddie has succeeded in getting Shed Skin 0.9 accepted into the Debian repositories, which is awesome for me personally.

Shed Skin 0.9

I have just released version 0.9 of Shed Skin, a restricted-Python to C++ compiler. Looking at the state of things, a 1.0 version is probably not far off. There will probably be several 0.9.x releases first, though.

There were two issues blocking a 1.0 release in my mind, both of which seem to have been fixed with this release.

First, I was not completely happy yet with the type inference scalability. With 0.9, there has been another dramatic improvement in this regard. For example, the C64 emulator is analyzed 10 times faster now (about 3,000 lines in 2 minutes on my PC). But most other example programs are now analyzed much faster. It now seems a realistic option to compile a 10,000 line program.

Second, there were some long standing nasty issues with combining 'incompatible' types (such as for '[[1.0]] == [[1]]', we are comparing different types). Shed Skin 0.9 includes a new framework for dealing with these issues, and the problem seems mostly solved at this point.

There were also some very nice performance optimizations for this release. Francois Boutines greatly optimized file I/O. Complex numbers are now copy-by-value, hence massively faster. And the idiomatic construct 'for .., .. in somedict.iteritems()' should be a lot faster as well now. Thanks to Andreas van Cranenburgh for triggering the latter (as well as some of the type inference improvements) with a nice new example program, a natural language parser.

Shed Skin 0.9 also supports three new modules. Francois Boutines added support for the 'mmap' module. Fahrzin Hemmati added support for 'binascii'. And Tony Veijalainen added support for 'colorsys'.

Further, there has been a massive amount of fixes. Thanks to Jason Ye for triggering several of these. Brent Pedersen fixed some 'set' issues. Francois Boutines fixed a string comparison bug involving \0, and improved 'universal mode' for files. Joris van Zwieten sent in the magic code to only print tracebacks (shedskin -x) for uncaught exceptions.

As usual, more details can be found in the release notes.

In the meantime, Paul Boddie has been at work trying to get Shed Skin 0.9 accepted as an official Debian package. It is already included with Fedora (thanks to Thomas Spura) and some other distributions, but I'm a Debian/Ubuntu user myself, so I would be really pleased to see this happen.

Shed Skin 0.8, Programming Language Benchmark

I'm happy to announce the release of Shed Skin 0.8, a (restricted-)Python-to-C++ compiler. It must be one of the best releases so far, with help and feedback from many directions. I'm especially thrilled to see an updated version of the C64 emulator example run one of my all-time favorite games.. :-) The emulator is now well over 3,000 lines (sloccount), and still compiling fine every time. The analysis time is in line with the scalability graph I published earlier. The game goes from a few FPS using CPython to a fluent 50 FPS after compilation on my PC.



The emulator triggered me to finally add support for the 'struct' and 'array' modules to Shed Skin. The struct module is now used to load programs from tape, and the array module is used to maintain a packed version of the screen data.

Another interesting new feature, also triggered by the emulator, allows generated code to print tracebacks for every raised exception (shedskin -x). So it's not entirely the same as in CPython (yet), which only does this for uncaught exceptions, but it's a step in the right direction. Unfortunately, I discovered at the last moment this doesn't work in the windows version. Hopefully this can be fixed for 0.9.

More details about the release can be found in the release notes.

Before I could make this announcement, there was a small performance comparison published, comparing different languages and implementations (such as PyPy, Jython and Shed Skin 0.8) for 4 different benchmarks. Unfortunately, Shed Skin 0.8 fails on one, but the problem is already fixed in GIT (regular expression search on empty string). The performance on the dict test can probably be attributed to the currently still slow I/O in Shed Skin. Improving this will be one of the goals for 0.9. Hopefully by the time you read this the comparison will have been updated at least for the failing test.

update: The file I/O bottleneck was fixed (thanks francois), and the performance comparison has been updated. Note that the first two benchmarks can still run a lot faster when disabling index-out-of-bounds checking (shedskin -b). On my PC, they become 30% and 45% faster, respectively..

update 2: the glitches are now also gone from the C64 example in GIT.

Shed Skin 0.7.1, Help needed!

I have just uploaded the tarball for Shed Skin 0.7.1, which fixes several important problems in 0.7, and adds a few minor features. The Windows version and hopefully the Debian/Fedora packages will follow shortly. As usual, please see the release notes for an overview of all the changes.

There aren't any major new features, because I haven't received much feedback since 0.7. Please let me know if you run into any problem, so it can be fixed, or if you know of some interesting program I could play with! I'd also like to take this opportunity to ask for some help in specific areas.

Excitingly, Thomas Spura was able to get Shed Skin 0.7 accepted in the Fedora repositories! :D Thanks to Thomas for that! Inspired by this, I'd like to hear from anyone who might be interested in assisting/motivating Paul Boddie to get his Debian package accepted in the Debian/Ubuntu repositories. I think if it got past the scrutiny of the Fedora people, this shouldn't be too hard.. :-)

I would also really appreciate to see some description of Shed Skin on Wikipedia (again), because I think it deserves this, especially since the scalability improvements in 0.6. But also of course because it sends lots of users to the homepage. If you'd like to work on this, please see this thread with links to independent blog entries and even a book that describe or benchmark Shed Skin. For a summary of how the type inference part works, please see my recent 0.7 announcement.

Since a while, inspired by the LibreOffice project, there has also been a page in the Shed Skin wiki that lists some easy tasks that one could try and tackle, to hopefully make it easier to start contributing to Shed Skin. I try hard not to fix these problems myself, in the hope someone else will pick them up.

update: I just added another great new example to the 0.7.1 tarball, a quantum monte carlo simulator by Mark Dewing. shedskin takes about 10 minutes on my PC to analyze its 1,200 lines of code (sloccount).

Shed Skin 0.7, Type Inference Scalability

I have just released version 0.7 of Shed Skin, a (restricted) Python-to-C++ compiler. The new version comes with lots of minor fixes and some optimizations, as well as a new Windows package. The Windows package was upgraded to a recent version of MinGW, which means it now includes GCC 4.5. I hope to package each new release for Windows again from now on.

There are also two nice new examples, one of which is over 1,000 lines (sloccount), for a total of 52 examples. Both are ray tracers, and it's always nice to look at the output of ray tracers. The larger one has a GUI and uses the multiprocessing module in combination with a Shed Skin generated extension module to do its heavy lifting. The other one uses a very slow but more realistic ray tracing technique, for a very nice result. Thanks to Eric Uhrhane and Jonas Wagner for sharing their programs!

I'd like to take the opportunity to talk a bit about type inference scalability. This graph shows the time Shed Skin needs for each of the current examples (seconds versus sloccount; the point on the right is the C64 emulator). I have to mention that two examples are left out here, because of a non-termination issue - that is, Shed Skin aborts after a certain amount of iterations. The reason I haven't looked into fixing this issue yet is that they do compile and run fine after Shed Skin aborts, because the type analysis is already precise enough at this point (though it takes some time). Precisely because of that, I don't think this issue would be hard to fix.

So while the current type analysis is obviously not perfect, I think this graph shows that type inference may not be all that hard as some people would have thought, at least for statically restricted code (the restriction does make it easier!). And I'm sure a team of programmers or someone smarter than me could do a much better job.

For those with some knowledge of the type inference literature, I'd like to explain at this point how Shed Skin achieves this scalability. Who knows this might be useful to people starting out in this area. It is actually quite simple conceptually (but don't think you can implement this correctly in a few weeks or even months for a large language such as Python!).

Shed Skin combines Ole Agesen's Cartesian Product Algorithm (CPA) with the data-polymorphic part of John Plevyak's Iterative Flow Analysis (IFA), both published in 1996 if I'm correct, and as suggested by Agesen. Because the CPA algorithm has a the tendency to explode for larger programs (usually in combination with the first, very imprecise iteration of IFA), Shed Skin 0.6 adds an incremental approach to the mix. It simply repeats its analysis for increasingly larger versions of a program, starting at nothing, essentially, and slowly adding to the callgraph until the whole program is analyzed. This seems to greatly avoid the CPA from exploding.

Shed Skin 0.6

I have just released version 0.6 of Shedskin, an experimental (restricted-)Python-to-C++ compiler.

Most importantly, this release finally comes with a substantial scalability improvement. Instead of analyzing a whole program at once, Shedskin now repeatedly analyzes an ever increasing version of the program, starting at nothing, essentially, until the whole program is analyzed. It turns out it this is much easier than analyzing programs as a whole. The result is that Shedskin now easily scales to several test programs of over 1,000 lines (sloccount), and needs much less memory to analyze them.

I'd really like to receive programs that Shedskin 0.6 is unabe to analyze within a reasonable amount of time, to get an idea of the actual extent of the improvement, and to be motivated to further improve things. It's hard to be motivated to improve things if all programs you know of work well!

Interestingly, because of the incremental analysis, it also became possible to add a progress bar to Shedskin. It will slow down as we go to 100 percent, and be somewhat inaccurate if parts of a program turn out to be unused, but I for one couldn't be happier with it.

Another area that has seen substantial improvements is the support for generating extension modules. For example, basic (un)pickling of compiled classes should now work, and inheritance and overloading should now be better exposed to the outside world.

The most important bugfix is probably to correct the module/linenumber information for warnings and errors. It turns out the '#{' feature messed up line numbers, and with code spanning several files, Shedskin would often display the wrong module as well. This should all be fixed now.

More details about the release can be found in the release notes.

I would like to thank the following people for contributing to this release: Hakan Ardo, Thomas Spura, Eric Uhrhane, Saravanan Shanmugham, HartsAntler and Douglas McNeil, as well as Plonjers, for sending in the nice test case that triggered the scalability improvement.

update: Just after the release, Danny Milosavljevic sent in a great 50th example program, a Commodore 64 emulator! The emulator itself is a work in progress, but it boots and you can start typing commands. After modifying it a bit so it compiles out-of-the-box with 0.6, I added it to the 0.6 example tarball.

Shed Skin 0.5

I have just released Shed Skin 0.5, an optimizing (restricted) Python-to-C++ compiler. Looking back at the release notes, I'm happy to see contributions from more people than ever. I'd also like to thank David Ripton again, for sending me his Project Euler solutions (see my previous post), and Jason Ye, for sending in so many issues.

The new release supports 64-bit integers (shedskin -l), which was necessary to get many of the Project Euler solutions to work. I'm wondering if this perhaps should become the default, since by default we want to stay as close to Python as possible.. But I don't think it's really useful that often, and it will probably be slower in many cases, so we'll see.

In comparing the Project Euler solutions with Psyco, it became clear that Shed Skin did not perform very well for certain types of iteration. I found out that exceptions in C++ are extremely slow, greatly slowing down e.g. set iteration. The new release avoids throwing StopIteration most of the time, that is, inside the builtins.

After basing the set implementation on CPython before, FFAO saw my invitation to do the same for the dict implementation, and quickly implemented it. I think Shed Skin was able to beat Psyco on about 5 more euler solutions with this new implementation.

Random number generation (without using shedskin -r) should also be much faster now.

Andy Miller sent in patches to add basic support for MSVC (shedskin -v). Although of course there is still no recent Windows package for Shed Skin. If anyone would like to volunteer, it should be easy to base a 0.5 package on the 0.3 version for Windows, though it would be nice to use a more recent version of MinGW.

Douglas McNeil sent in patches to add support for __future__.print_function and generator methods (as opposed to non-method functions). Thomas Spura optimized printing somewhat, though there is still some room for optimization there. Finally, Michael Elkins sent in some improvements for the socket module (which he originally wrote).

More details about the new release can be found in the release notes.

Shedskin versus Psyco for 131 Project Euler Solutions

(Shedskin is an experimental (restricted) Python-to-C++ compiler.)

After releasing 0.4 I received a link to a blog post comparing PyPy, Unladen Swallow and Psyco for a set of over 100 project euler solutions. Of course, I decided to ask the author for the source code of these solutions, to have a go with Shedskin. It turned out to be a great test set for spotting performance flaws, and it even convinced me to add 64-bit integer support (shedskin -l), though I'm still not sure how generally useful that is.

Out of 204 solutions in total, I was able to get 131 to compile and run correctly (using the new 64-bit support), at a total of about 5,000 lines (sloccount). In some cases, I had to work around a Shedskin limitation, such as that mixing None and integers or nested functions are not allowed, but these were all very simple changes. While in many cases, it would have been easy to optimize things for Shedskin, I didn't make a single change other than to get things to compile. The other 73 solutions either needed arbitrary-precision arithmetic or use some unsupported module (mostly 'gmpy', whatever that is).

After optimizing Shedskin here and there (though only in generally useful ways, nothing really specific to these tests), I compared the run-time of Shedskin with that of Psyco on these 131 solutions. The graph below shows the relative speedup of Shedskin versus Psyco, taking the shortest time of three runs for each. Shedskin is faster in 109 cases out of 131, and if we take the cumulative time (730 versus 1330 seconds), 45% faster in total. Many of the 22 cases where Psyco wins deal with dictionaries, and I think Shedskin could also win most of these with a better dict implementation (any volunteers?).



A reason why the results may be somewhat flattering for Shedskin, is that quite a few solutions run in only a fraction of a second, and Psyco may suffer from a longer startup time (though I wouldn't know). On the other hand, many of the solutions are quite memory-intensive and spend a lot of time in the python builtins, so Shedskin often doesn't get a chance to shine. I think with a few manual changes to the solutions, many would run much faster using both Psyco and Shedskin, but with a larger speedup for Shedskin.

Unfortunately, Project Euler being semi-competitive and all, I cannot share the solutions. But I will mail the timings for each solution to the discussion group. Anyway, here's the setup I used to calculate the graph:

- shedskin -l (SVN 2010-5-1)
- g++ 4.4.3 -O3 -fprofile-generate/use -march=native -mtune=native -fomit-frame-pointer
- q9550s CPU
- each solution is run three times with Psyco, then three times with Shedskin, with a time.sleep(2) after each run
- the fastest time for each is taken.

So again, if anyone would be interested in rewriting the dict implementation, based on the CPython code (instead of relaying to the STL), that would be great. FFAO did this before for the set implementation, and that helped a lot.

Shed Skin 0.4

I have just released Shed Skin 0.4, an experimental (restricted-)Python-to-C++ compiler. Thanks again especially to Jeremie Roquet for helping out.

The biggest 'improvement' of this release is perhaps that Windows is no longer supported. Windows users are encouraged to take over maintainership of the MinGW version, or to upgrade to a 'real' operating system, such as Ubuntu, and live in software freedom.

Other improvements include support for generator expressions, real boolean support (instead of integers that print as 0 and 1), support for the 'key' argument of 'min' and 'max', another useful type inference scalability improvement, and support for heapq.{merge, nsmallest, nlargest}. Please see the release notes for the full list of changes.

Four new example programs were also added, for a total of 47 programs:

-pylife, a game of life implementation based on the wonderful hashlife algorithm (David Bau)
-a nice a-star implementation, with a pygame frontend (John Eriksson)
-a simple game of life implementation that gave the type analysis some trouble (Francesco Frassinelli)
-a second genetic algorithm (Stavros Korokithakis)

Please go ahead and try it out, and let me know about any problems. I'm also always very happy to receive new example/test programs to play with, especially if type inference fails (or doesn't terminate) for them!

Shed Skin 0.3

After 5 years of work, I'm proud to announce Shed Skin 0.3, an experimental (but obviously restricted) Python-to-C++ compiler. Looking over the release notes, I'm convinced this must be the best release so far. I would like to thank especially Jeremie Roquet, for several major contributions to this release, and Thomas Spura, for reorganizing the codebase. Joris van Rantwijk provided the great new 'maximum weighted matching' example, and he and Mike Schrick sent in minor but useful patches.

For me, these are the highlights of the release:

- support for 'itertools', 'heapq' and 'csv', bringing the total set of (mostly) supported library modules to 20
- 4 new example/test programs, bringing the total to 44 examples, at over 10,000 lines in total (sloccount)
- improved type inference scalability (while greatly simplifying the code)
- support for 'map', 'filter' and 'reduce' (yes, finally, go wild.. :-))
- support for the 'with' statement
- support for the 'key' argument to 'sorted' (and 'list.sort')
- reduced the compiler core by about 100 lines (again)
- a distutils-based setup.py (talk about 'finally'!)
- several important optimizations (indexing, slicing, adding 1-length sequences)
- an unusually large number of bug fixes

Please go ahead and try it out, and start sending in those issues. I'm also always very happy to receive new example/test programs to play with, especially if type inference fails (or doesn't terminate) for them!

Shedskin: what next?

(Shed Skin is an experimental (restricted) Python-to-C++ compiler.)

After playing with Disco for a while, and after that being away for over a month, I'm ready to put some time in Shed Skin again. I've just applied a few patches that I received (thanks!!), fixed some minor issues myself and added a few optimizations. But I'm not really sure which 'big' thing to work on next, and I don't really know of a nice program to test with at the moment (such as Minilight or Disco). I'd love to hear some suggestions!

update: I posted a feature plan for 0.2.1 on the mailing list. thanks for the feedback!

Disco: an elegant Python Go player (update)

after receiving generous help from 'apt1002' in the datastructure department, I am happy to announce Disco 0.3 (see my previous posting about Disco).

the new version is about ten times faster than 0.2 (from around 600 games per second on my PC to about 6000). in addition it now checks for positional superko (repeated positions), so it can now play on CGOS without making illegal moves half of the games.

interestingly, using CPython there is not much difference in speed between 0.2 and 0.3, while the speedup from using Shedskin goes from about 6 to 75 times! this is probably mostly due to avoiding (re)allocations, which of course aren't much faster in C++ than in CPython.. but the new datastructures are also a bit smarter in maintaining only the most necessary information.

the sourcecode is still only about 400 lines, but now slightly less readable.. I hope to clean this up a bit over the next few months, but other than I will be too busy. in the long-term, it would be interesting to add shape analyses and additions to UCT such as RAVE/AMAF, and try to keep all of that in a pretty package of under 1,000 lines..

Shed Skin 0.2

I have just released version 0.2 of Shed Skin, an experimental (restricted) Python-to-C++ compiler. It comes with 7 new example programs (for a total of 40 example programs, at over 12,000 lines) and several important improvements/bug fixes. See here for the full changelog.

The new example programs consist of Disco, an elegant go player (see my previous blog post), a larger Voronoi implementation at 800 lines, a TSP algorithm simulating ant colonies, a nicer neural network algorithm and three compressors (Lempel-Ziv, huffman block, and arithmetic).

Other than bug fixes for these programs, this release adds some important optimizations. First and foremost, inlining was greatly improved, resulting in potential speedups across the board. Second, loops such as 'for a, b in enumerate/zip(sequence[, sequence])' should now be dramatically faster (also inside list comprehensions), by avoiding allocation of intermediate tuples. Finally, basic list slicing should now be much faster.

Disco: an elegant Python Go player

There's this shiny new technique to implement computer Go players, called Monte Carlo UCT. It shocked the Go world by powering the first computer player (Mogo) to beat (quite unexpectedly I think) a professional player in an official match (with only a 7 stone handicap). Ever since hearing about this rather simple new technique, my hands have been itching to implement it in Python and compile the result with Shedskin.

The name of the resulting Go player is Disco (think "staying alive"), and its source code of only 350 lines can be found here. Given enough time, it can play some pretty interesting games on a 9x9 board. And of course you can connect it to a graphical Go board, although I only tested it with Gogui. See the README for installation and configuration details.

It has become quite a nice test case/example for Shedskin. Not only does it become about 5 times faster on my computer after compilation, and shows that you can write a pretty fast Go engine using it, it also shows how easy it is to generate an extension module with Shedskin and effectively use this in a larger program. In this case, there is a Parallel Python wrapper that fires up an abitrary number of processes, each importing and using the extension module to do part of the thinking.

Sometimes everything can come so beautifully together: an elegant Python program of only 350 lines, compiled to an extension module, imported in multiple Python processes, connected to a graphical Go board and playing according to some brilliant but simple new mathematical formula that can somehow make computers play Go! Now if only I would be better at Go myself, so I could really test it..

BTW, Shedskin 0.1.2 should be right around the corner. Obviously I got a bit distracted by the above.. In all, it contains many optimizations and bug fixes and adds 5 other new example programs.

Shed Skin 0.1.1

I have just released version 0.1.1 of Shed Skin, an experimental (restricted) Python-to-C++ compiler. It comes with 5 new example programs (for a total of 35 example programs, at over 10,000 lines) and several important improvements/bug fixes. See here for the full changelog.

The most interesting new example is minilight, an elegant raytracer (more precisely, a global illumination renderer) that utilizes triangle primitives and an octree spatial index. As shown on the minilight homepage, it becomes up to 100 times faster.

Other new examples are Peter Norvig's sudoku solver (which unfortunately doesn't become faster, but is a cool example anyway), yet another raytracer and a mastermind strategy evaluator by Raymond Hettinger.

The fifth new example, an interactive circle packing program, is especially nice, because it is, well, interactive, but also because it shows how easy it is to generate an extension module with Shed Skin and use this in a larger program (a Pygame application in this case).

The biggest improvement (if you can call it this) for this release was to drop support for generic types. This simply means that functions that can be called with different kinds of types of objects for a single argument are no longer supported (unless these types have a common base-class, of course). Similarly, generic datastructures are no longer supported. Dropping support for generic types made the compiler core a lot simpler (and 10% smaller!), and removed the need for the -i command-line option.

Seeing the compiler core shrink so much, I was inspired to further refactor several messy areas of the compiler for readability (most notably in infer.py). This means the compiler core has also become a lot more readable/hackable with this release. I hope to continue this work of refactoring (and adding docstrings) with each release from now on, and invite anyone to help out here.

In general, I would really like to receive more help. For example, if someone could take over maintainership of the Windows version (and possible upgrade the MingW distribution that is packaged with it), that would be great. A suggestion on the coding side might be to add keyword support to generated extension modules (see extmod.py), or to optimize some builtins (for example, string/list slicing, see lib/builtin.?pp). But I would also be happy to receive more interesting test cases (most come from a single person at the moment) and/or more bug reports (which I get far too few of.)

Minilight Compiled

(Shed Skin is an experimental (restricted-)Python-to-C++ compiler.)

Minilight is an elegant minimal global illumination renderer, or raytracer, that uses triangle primitives and an octree spatial index. The original version consists of about 1,000 lines of C++, but there are several translations available at the homepage, such as ~500 line OCaml and Python versions. The Python version apparently clocks in 173 times later than the C++ version, while the OCaml version is only about 3 times slower.. Wouldn't it be interesting if we could make the Python version run faster than the OCaml version?

After some small changes (splitting up some dynamic variables, replacing calls to 'map', 'type'..), and applying some minor fixes to Shedskin, I was able to compile the Python version to C++ using Shedskin SVN. For the basic Cornell box example, it runs about 35 times faster than the modified Python version (which should again be a bit faster than the original Python version). It's probably still somewhat slower than the OCaml version, but not that much. To try to compile it yourself, you'll probably want to have more than 512 MB of RAM, and use the 'shedskin -i' option. The modified version can be found in /examples/minilight in SVN.

I'm guessing that the Python version can be made a bit faster in itself. What typically happens then is that the C++ version will get relatively even faster, because certain bottlenecks that don't really slow down Python will slow down C++ (such as allocating lots of objects). If you'd like to try and help beat OCaml, I'd be very interested in hearing about potential improvements (where readability doesn't suffer too much, obviously.)

Another program that I managed to compile, while much smaller, is one that solves a well-known chess puzzle. To compile knight2.py, I had to replace the 'key' argument to 'sorted' with a 'cmp' version (as 'key' is not yet supported), and fix a small bug in Shedskin ("'%*d' % (2, 7)" and such did not work yet.)

I'm always interested in hearing about other interesting test cases for Shedskin, as they seem hard to come by. Please also consider sending in bug reports. Both types of feedback are essential for me to keep working on Shedskin!

Update: After some tweaking, it is now more than 60 times faster than the original here. See the comments for more information.

Update 2: I removed the -i option with Shedskin 0.1.1, so it is not necessary anymore.