Sunday, July 19, 2009

Shed Skin 0.2

I have just released version 0.2 of Shed Skin, an experimental (restricted) Python-to-C++ compiler. It comes with 7 new example programs (for a total of 40 example programs, at over 12,000 lines) and several important improvements/bug fixes. See here for the full changelog.

The new example programs consist of Disco, an elegant go player (see my previous blog post), a larger Voronoi implementation at 800 lines, a TSP algorithm simulating ant colonies, a nicer neural network algorithm and three compressors (Lempel-Ziv, huffman block, and arithmetic).

Other than bug fixes for these programs, this release adds some important optimizations. First and foremost, inlining was greatly improved, resulting in potential speedups across the board. Second, loops such as 'for a, b in enumerate/zip(sequence[, sequence])' should now be dramatically faster (also inside list comprehensions), by avoiding allocation of intermediate tuples. Finally, basic list slicing should now be much faster.

21 comments:

philhassey said...

cool :)

Lucian said...

Very nice.

I've been thinking, could you use function annotations (http://www.python.org/dev/peps/pep-3107/) to help the type inference?

srepmub said...

probably, yes. but I have a strong dislike for manual type declarations. I'd much rather get type information from a profiler.. a profiler can give the same and other useful information, without having to change or add anything to the code.

Lucian said...

I share that dislike, but type inference (especially in Python) doesn't scale very well. Manual declarations might help it along.

srepmub said...

type inference can be made to scale a lot better than shedskin currently does, for statically typed programs at least. I just put together some basic techniques from the literature as my master's thesis.. I'm sure a real TI expert could make it work much, much better. that said, if you dislike manual type declarations as well, then you should agree profiling is the way to go to get type information :)

Lucian said...

Yeah, I know. I was just thinking that perhaps helper type signatures are easier to implement.

You may want to look into how the PyPy guys are doing it, translating RPython to C/.NET/JVM

Ben McDonald said...

Really impressive and just the sort of project I was hoping someone was working on. Thank you! Good luck with the project.

Tonetheman said...

good stuff. shedskin rocks!

Seun Osewa said...

Just finished running some Shedskin tests. Your approach really blows Psyco and Cython away performance-wise.

And your packaging is excellent too - you included everything one needs to create Shedskin programs. (Once installed, just type "shedskin myfile.py" and "make" and you're good to go.)

I'm really impressed. Well done!

srepmub said...

thanks for all the nice comments!

please also consider sending in bug reports (I saw yours seun, thanks!!), and interesting test-cases (similar to the 40 example programs, preferrably on the big side), too help motivate me to keep improving shedskin.

Chuong said...

Shedskin is amazing! Please continue the great work.

I had an error when trying to run TonyJpegDecoder:

terminate called after throwing an instance of '__shedskin__::IndexError*'

Thanks,
Chuong

Chuong said...

PS: Can ShedSkin compile itself so that it (compiled ShedSkin) compiles python codes faster?

srepmub said...

hi chuong,

the python version also throws an IndexError, so this is correct.. :) (you have to provide it with a .jpg file to decode)

shedskin cannot compile itself, because it uses dynamic typing in places (for example, via the visitor pattern). it might be possible to compile just the type inference part, but I don't think it's worth the effort at the moment.

Mike said...

Great job, I love the fact that you're taking care of dynamic typing, this is going to be huge! I have modified your setup.py script to allow for installation on unix platforms if you're interested.

thanks

ArquerosNeuquinos said...

Hi!

I'm not a user of python but a openemebedded developer.

This metadistro uses bitbake, a "Portage like" tool writed in python and accelerated with psyco

There is some way to compile bitbake with shed skin?

srepmub said...

thanks for your question!

please see the shedskin tutorial for an overview of the limitations.

I'd also be happy to have a look at the source code, if you could just send it to me..

srepmub said...

@mike:

sure, thanks, please submit it as an issue at the googlecode site.

ArquerosNeuquinos said...

Here is the bitbake project

http://developer.berlios.de/projects/bitbake/

srepmub said...

I'm afraid it's way too big and dynamic (as described in the tutorial). it might be possible to compile some time-critical part as an extension module, but I wouldn't know which part.. :)

ArquerosNeuquinos said...

Well, I'm not an expert but the bottlenck in the process is when try to resolve any missing task queue dependencies.

The part of the code tha do that is taskdata.py and the asociated programs (data, event, mkdirhier, utils)

srepmub said...

if you send me a version of the code that should (almost) work according to the tutorial, I'd be happy to have a look..