Shed Skin: Call for participation

Okay, so I have built this pretty cool (restricted) Python-to-C++ compiler, that actually works for many not-too-large programs. It currently builds (simple) extension modules, and shows massive speedups for many programs, often outperforming Psyco by a factor. As I recently showed on my blog, it also integrates nicely with Parallel Python. I would have thought the latter would have given rise to at least one comment..

Am I the only one seeing the potential of an implicitly statically typed Python-like-language that runs at practically the same speed as C++? Is it that being ahead of the time makes you completely misunderstood, even by pretty smart hackers? I remember developing something quite like Wikipedia many years ago, and being unable to convince anyone of why this would be a good idea.. Of course that doesn't mean my compiler is a good idea (I wish), but I'm sure not going to give up as easily this time.

So, once again, I'd like to ask for more participation. Many open source users don't realize, I think, how open source projects, especially new ones, thrive on user feedback. Programming is debugging, and there's nothing more satisfying than fixing particular problems users encounter. Sure, I could find most bugs myself, but I don't want to end up in a nut house, and at some point there has to be some kind of community process. Patches are very useful too, even simple ones, as they often trigger more patches by me. Extension module support, for example, started out as a simple proof-of-concept patch sent in by a user.

I'd very much like to take Shed Skin further forward, but I need your help to do so! Please visit the homepage and send in bug reports or join the mailing list and start some discussions. I can also be hired to work on specific features of course :) So hurry to the homepage:

http://mark.dufour.googlepages.com

Shed Skin and Parallel Python

Shed Skin is an experimental Python to C++ compiler. Parallel Python allows for clean and simple parallelization of Python processes over multiple cores. Wouldn't it be cool if we could combine the two? Of course I wouldn't be writing this if I hadn't tried. Here's how to do it.

Create an extension module with Shed Skin, containing function(s) you'd like to use in parallel. For example, we might use the partial sum function from the Parallel Python website:

def part_sum(start, end):
..
return sum

For Shed Skin's type inference to work, part_sum must be called from somewhere:

if __name__ == '__main__':
part_sum(1, 1)

Creating an extension module is simple (suppose the module is named meuk.py):

ss -e meuk
make

Because Parallel Python expects pure-Python code, we must call our compiled function via a pure-Python wrapper:

def part_sum(start, end):
import meuk
return meuk.part_sum(start, end)

In order for Parallel Python to find our extension module (at least on my Ubuntu system), we must issue this in advance:

export PYTHONPATH=$pwd

And there you have it. Here are some timings:

no extension module, 1 worker: 11.3 seconds
no extension module, 2 workers: 6.2 seconds
extension module, 1 worker: 0.6 seconds
extension module, 2 workers: 0.3 seconds

Shed Skin 0.0.24, 0.0.25

I've just released Shed Skin 0.0.25. Together with the (unannounced) 0.0.24 release, there have been some interesting changes. Most importantly perhaps, Shed Skin now caches (most) 1-length strings, which can improve performance dramatically for string-intensive programs. I also performed a long-overdue rewrite of the virtual function detection code, which should work much more reliably now, at least for relatively simple cases :)

0.0.24:
-1-length string caching

0.0.25
-improved detection of virtual functions
-further set optimizations
-fix for extension modules and certain default arguments
-exhaustive checking of C++ keywords
-fix for some combinations of arguments to min, max
-several minor bug fixes

As always, I could really use more help in pushing Shed Skin forward. Let me know if you'd like to help out, but are not sure where to begin.

Shed Skin 0.0.23

I have just released Shed Skin 0.0.23. It doesn't contain the type inference scalability improvements I was working on, but it does have quite a few bug fixes and minor feature additions. Here's a list of changes:

-support for __iadd__, __imul__ and such (except __ipow__ and __imod__)
-some overdue set optimizations
-fix for string formatting problem (%% did not always work)
-extension module stability fixes
-fix for particular inheritance problem
-other minor bugfixes, cleanups, and error messages

I could really use some systematic help in pushing Shedskin further. Some ideas:

-send in bug reports - these are extremely valuable and motivating to me, yet I don't receive many..
-find out why test 148 is currently broken under windows
-add datetime, re or socket support
-look into supporting custom classes in generated extension modules
-write a Shedskin tutorial for 'novice' programmers
-systemically test performance and suggest and work on improvements
-investigate replacements for std::string and __gnu_cxx::hash_set
-perform janitorial-type work in ss.py and lib/builtin.?pp
-support extension modules under OSX (OSX gives me accute mental RSI)
-add more tests to unit.py

Shed Skin 0.0.22

after being bogged down with work for a few weeks, I got back to development again. the trigger was a proof-of-concept patch sent in by Harri Pasanen for generating extension modules. previously I was waiting for someone else to fully tackle this, but his patch was quite simple, and I found out it easily works under mingw, too. so I generalised things a bit, and released it as part of version 0.0.22. the process of building an extension module is now quite simply 'ss -e ..' and 'make' (note it doesn't work under OSX, yet).

there are some limitations though as to the way this works:

-only builtin scalar and containers can be passed/returned (int, float, str, list, tuple, dict, set)
-arguments/returned objects are completely copied/converted at call/return time (i.e., including contained objects)
-global variables are considered constant, and converted at module initialization time

consider this simple program, mod_name.py:

some_var = [1,2,3]

def some_func(x):
return 2*x

if __name__ == '__main__':
some_func(1) # obviously, this is needed for type inference to work

to compile this down to an extension module, simply use the new '-e' command-line parameter:

ss -e mod_name
make

that's it. now the program can be used from an arbitrary python program/prompt:

>>>import mod_name
>>>dir(mod_name)
>>>mod_name.some_var
>>>mod_name.some_func(1)

I'm hoping someone else can add support for custom classes and find out how to get this to work under OSX!

Shed Skin 0.0.20 and 0.0.21

development has been continuing at a steady pace, resulting in two new releases. I haven't gotten around to cleaning up and improving tuple support, as of course I got bogged down into other details. the first release essentially combines many minor fixes:

Release Name: 0.0.20
-improvements to aug-assignments with subscripting ('a[x, y] += 1' and such)
-fixed some problems with slice-assignments ('a[1:-1] = [1,2])
-make integer division (/,//,divmod,floordiv) equal to CPython for negative/positive combinations of arguments
-make printing of floats closer to CPython
-move generic functions/methods to header file
-many small fixes (allow 'self' as function argument, list.extend takes iterable, __delitem__ overloading, raw_input(), 'return' from generator..)
-improved error checking for dynamic types
-optimize expr**2 and expr**3 using __power2 and __power3 functions

the next release is more interesting. first, a problem exhibited in a little program bearophile sent me (I wish there were two bearophiles :-)) gave me the insight that two things I was doing during type inference were more or less the same; this allowed me to generalize things and cut away about 100 lines. second, I added support for 'bisect', 'collections.deque' and improved 'copy' support. finally, there were some important dict optimizations (dict[..] += .. and dict.__getitem__ now only index once):

Release Name: 0.0.21
-important type inference fix/cleanup
-support for 'bisect', 'collections.deque' and 'string.maketrans'
-improved 'copy' support
-support for 'try, else' construction
-some optimizations ('dict[..] += ..', "''.join(sorted(str))")
-several minor bug fixes

Shed Skin 0.0.19

I have just released 0.0.19, with some interesting changes. Iterators and generators are supported now. Full random support was contributed by a user (thanks Jeff :-)), by converting a Python version to C++ using Shedskin. Interestingly, by default it uses the same random engine (Mersenne Twister) as CPython, so the two behave exactly the same. Jeff is now thinking about how to add fast matrix support, which would be really nice to have.. But there were also many small fixes in this release. Here is the sourceforge changelog:

-iterator and generator support (no generator expressions yet)
-full support for random module (converted from Python version)
-added itertools.{count, repeat, cycle}, dict.{iterkeys, itervalues, iteritems}
-added 3-argument integer version of __builtin__.pow
-logical and, or operations have correct return value now (e.g., '3 and 4')
-lots of minor bug fixes (improved % operator, added math.pow..)

The main focus for 0.0.20 will probably be to add support for tuples, with differently typed elements, of length greater than 2, up to some arbitrary length..

Another Raytracer

here's the output of a raytracer in python, called yopyra, by carlos gonzales, after being compiled by shedskin. the speedup is about the same as for the pygmy raytracer, that is part of the unit tests (for this picture it is about 60). I will probably add this program to ss-progs, but I need to remove one hack to support it, and bearophile (who sent me the raytracer) is still adding some nice features, like the ability to read scenes from a file, and render parts of a picture, so it's easy to put those multiple cores to use..

http://mark.dufour.googlepages.com/scene.txt.jpg

Shed Skin 0.0.18

yes, it's the fourth release of shed skin in about a month. can you tell I'm working on it full-time now? :) here's a list of major changes:

-modifications to work on OSX and 64-bit systems
-improved support for class attributes
-classes can now be used prior to their definition
-several optimizations and fixes for strings
-hash value caching for strings and tuple{int,double,str}
-optional bounds checking using --bounds

thanks to jplevyak, larry, gustavo and denis for helping to get SS to work on 64-bit and OSX systems!

note that it's very hard for CPython to cache hash values for tuples (because the elements might be mutable). once types are known however, e.g., for tuple, SS can cache hash values no problem. this can cause quite a speedup.. for example, the following program is now 10 times faster than under CPython (corresponding to the 10 loop iterations):

sign_words = {}
kawamabahana = 10000000*(1,2)
for x in range(10):
hash(kawamabahana)

I'm going to take a few days off now, and think about a strategy to add iterators (and hence generators and generator expressions), and possibly non-uniform tuples of length greater than 2..

24 programs that work

I'm collecting programs that work out-of-the-box with Shedskin CVS, to add to the homepage before releasing 0.0.18 (which should be out within the next few days). So far, I have 24 programs, at a total of about 6,000 lines:

http://mark.dufour.googlepages.com/ss-progs.tgz

Please let me know if you have an interesting program I might add to this set. If someone else would like to perform some testing (analysis time and comparing performance with CPython and Psyco, for example), that would be great.

update: __tim__ pointed me to the nbody debian language shootout test, which becomes about 100 times faster here. I added it to ss-progs.

Shed Skin 0.0.17

And here goes 0.0.17, with some interesting fixes:

-support for assignment lists, so e.g. '[[a,b],[c,d]] = matrix' and 'for [[a,b],[c,d]] in matrices: ..' work now
-'_' can be used as special assignment target (expression is not assigned), e.g. '[[a,_],_] = matrix' works now
-support for '__name__' attribute of modules, so e.g. "if '__name__' == '__main__'" works now
-various improvements in exception handling (AssertionError, except some_tuple, ..)
-limit on tuple/list unpacking removed, so e.g. a,b,c,d,e,f = some_tuple works now
-improved printing of floats (using "%g" instead of "%f")