Shed Skin and Parallel Python

Shed Skin is an experimental Python to C++ compiler. Parallel Python allows for clean and simple parallelization of Python processes over multiple cores. Wouldn't it be cool if we could combine the two? Of course I wouldn't be writing this if I hadn't tried. Here's how to do it.

Create an extension module with Shed Skin, containing function(s) you'd like to use in parallel. For example, we might use the partial sum function from the Parallel Python website:

def part_sum(start, end):
..
return sum

For Shed Skin's type inference to work, part_sum must be called from somewhere:

if __name__ == '__main__':
part_sum(1, 1)

Creating an extension module is simple (suppose the module is named meuk.py):

ss -e meuk
make

Because Parallel Python expects pure-Python code, we must call our compiled function via a pure-Python wrapper:

def part_sum(start, end):
import meuk
return meuk.part_sum(start, end)

In order for Parallel Python to find our extension module (at least on my Ubuntu system), we must issue this in advance:

export PYTHONPATH=$pwd

And there you have it. Here are some timings:

no extension module, 1 worker: 11.3 seconds
no extension module, 2 workers: 6.2 seconds
extension module, 1 worker: 0.6 seconds
extension module, 2 workers: 0.3 seconds

Shed Skin 0.0.24, 0.0.25

I've just released Shed Skin 0.0.25. Together with the (unannounced) 0.0.24 release, there have been some interesting changes. Most importantly perhaps, Shed Skin now caches (most) 1-length strings, which can improve performance dramatically for string-intensive programs. I also performed a long-overdue rewrite of the virtual function detection code, which should work much more reliably now, at least for relatively simple cases :)

0.0.24:
-1-length string caching

0.0.25
-improved detection of virtual functions
-further set optimizations
-fix for extension modules and certain default arguments
-exhaustive checking of C++ keywords
-fix for some combinations of arguments to min, max
-several minor bug fixes

As always, I could really use more help in pushing Shed Skin forward. Let me know if you'd like to help out, but are not sure where to begin.