after receiving generous help from 'apt1002' in the datastructure department, I am happy to announce Disco 0.3 (see my previous posting about Disco).
the new version is about ten times faster than 0.2 (from around 600 games per second on my PC to about 6000). in addition it now checks for positional superko (repeated positions), so it can now play on CGOS without making illegal moves half of the games.
interestingly, using CPython there is not much difference in speed between 0.2 and 0.3, while the speedup from using Shedskin goes from about 6 to 75 times! this is probably mostly due to avoiding (re)allocations, which of course aren't much faster in C++ than in CPython.. but the new datastructures are also a bit smarter in maintaining only the most necessary information.
the sourcecode is still only about 400 lines, but now slightly less readable.. I hope to clean this up a bit over the next few months, but other than I will be too busy. in the long-term, it would be interesting to add shape analyses and additions to UCT such as RAVE/AMAF, and try to keep all of that in a pretty package of under 1,000 lines..
Sunday, August 30, 2009
Sunday, July 19, 2009
Shed Skin 0.2
I have just released version 0.2 of Shed Skin, an experimental (restricted) Python-to-C++ compiler. It comes with 7 new example programs (for a total of 40 example programs, at over 12,000 lines) and several important improvements/bug fixes. See here for the full changelog.
The new example programs consist of Disco, an elegant go player (see my previous blog post), a larger Voronoi implementation at 800 lines, a TSP algorithm simulating ant colonies, a nicer neural network algorithm and three compressors (Lempel-Ziv, huffman block, and arithmetic).
Other than bug fixes for these programs, this release adds some important optimizations. First and foremost, inlining was greatly improved, resulting in potential speedups across the board. Second, loops such as 'for a, b in enumerate/zip(sequence[, sequence])' should now be dramatically faster (also inside list comprehensions), by avoiding allocation of intermediate tuples. Finally, basic list slicing should now be much faster.
The new example programs consist of Disco, an elegant go player (see my previous blog post), a larger Voronoi implementation at 800 lines, a TSP algorithm simulating ant colonies, a nicer neural network algorithm and three compressors (Lempel-Ziv, huffman block, and arithmetic).
Other than bug fixes for these programs, this release adds some important optimizations. First and foremost, inlining was greatly improved, resulting in potential speedups across the board. Second, loops such as 'for a, b in enumerate/zip(sequence[, sequence])' should now be dramatically faster (also inside list comprehensions), by avoiding allocation of intermediate tuples. Finally, basic list slicing should now be much faster.
Thursday, July 09, 2009
Disco: an elegant Python Go player
There's this shiny new technique to implement computer Go players, called Monte Carlo UCT. It shocked the Go world by powering the first computer player (Mogo) to beat (quite unexpectedly I think) a professional player in an official match (with only a 7 stone handicap). Ever since hearing about this rather simple new technique, my hands have been itching to implement it in Python and compile the result with Shedskin.
The name of the resulting Go player is Disco (think "staying alive"), and its source code of only 350 lines can be found here. Given enough time, it can play some pretty interesting games on a 9x9 board. And of course you can connect it to a graphical Go board, although I only tested it with Gogui. See the README for installation and configuration details.
It has become quite a nice test case/example for Shedskin. Not only does it become about 5 times faster on my computer after compilation, and shows that you can write a pretty fast Go engine using it, it also shows how easy it is to generate an extension module with Shedskin and effectively use this in a larger program. In this case, there is a Parallel Python wrapper that fires up an abitrary number of processes, each importing and using the extension module to do part of the thinking.
Sometimes everything can come so beautifully together: an elegant Python program of only 350 lines, compiled to an extension module, imported in multiple Python processes, connected to a graphical Go board and playing according to some brilliant but simple new mathematical formula that can somehow make computers play Go! Now if only I would be better at Go myself, so I could really test it..
BTW, Shedskin 0.1.2 should be right around the corner. Obviously I got a bit distracted by the above.. In all, it contains many optimizations and bug fixes and adds 5 other new example programs.
The name of the resulting Go player is Disco (think "staying alive"), and its source code of only 350 lines can be found here. Given enough time, it can play some pretty interesting games on a 9x9 board. And of course you can connect it to a graphical Go board, although I only tested it with Gogui. See the README for installation and configuration details.
It has become quite a nice test case/example for Shedskin. Not only does it become about 5 times faster on my computer after compilation, and shows that you can write a pretty fast Go engine using it, it also shows how easy it is to generate an extension module with Shedskin and effectively use this in a larger program. In this case, there is a Parallel Python wrapper that fires up an abitrary number of processes, each importing and using the extension module to do part of the thinking.
Sometimes everything can come so beautifully together: an elegant Python program of only 350 lines, compiled to an extension module, imported in multiple Python processes, connected to a graphical Go board and playing according to some brilliant but simple new mathematical formula that can somehow make computers play Go! Now if only I would be better at Go myself, so I could really test it..
BTW, Shedskin 0.1.2 should be right around the corner. Obviously I got a bit distracted by the above.. In all, it contains many optimizations and bug fixes and adds 5 other new example programs.
Wednesday, April 22, 2009
Shed Skin 0.1.1
I have just released version 0.1.1 of Shed Skin, an experimental (restricted) Python-to-C++ compiler. It comes with 5 new example programs (for a total of 35 example programs, at over 10,000 lines) and several important improvements/bug fixes. See here for the full changelog.
The most interesting new example is minilight, an elegant raytracer (more precisely, a global illumination renderer) that utilizes triangle primitives and an octree spatial index. As shown on the minilight homepage, it becomes up to 100 times faster.
Other new examples are Peter Norvig's sudoku solver (which unfortunately doesn't become faster, but is a cool example anyway), yet another raytracer and a mastermind strategy evaluator by Raymond Hettinger.
The fifth new example, an interactive circle packing program, is especially nice, because it is, well, interactive, but also because it shows how easy it is to generate an extension module with Shed Skin and use this in a larger program (a Pygame application in this case).
The biggest improvement (if you can call it this) for this release was to drop support for generic types. This simply means that functions that can be called with different kinds of types of objects for a single argument are no longer supported (unless these types have a common base-class, of course). Similarly, generic datastructures are no longer supported. Dropping support for generic types made the compiler core a lot simpler (and 10% smaller!), and removed the need for the -i command-line option.
Seeing the compiler core shrink so much, I was inspired to further refactor several messy areas of the compiler for readability (most notably in infer.py). This means the compiler core has also become a lot more readable/hackable with this release. I hope to continue this work of refactoring (and adding docstrings) with each release from now on, and invite anyone to help out here.
In general, I would really like to receive more help. For example, if someone could take over maintainership of the Windows version (and possible upgrade the MingW distribution that is packaged with it), that would be great. A suggestion on the coding side might be to add keyword support to generated extension modules (see extmod.py), or to optimize some builtins (for example, string/list slicing, see lib/builtin.?pp). But I would also be happy to receive more interesting test cases (most come from a single person at the moment) and/or more bug reports (which I get far too few of.)
The most interesting new example is minilight, an elegant raytracer (more precisely, a global illumination renderer) that utilizes triangle primitives and an octree spatial index. As shown on the minilight homepage, it becomes up to 100 times faster.
Other new examples are Peter Norvig's sudoku solver (which unfortunately doesn't become faster, but is a cool example anyway), yet another raytracer and a mastermind strategy evaluator by Raymond Hettinger.
The fifth new example, an interactive circle packing program, is especially nice, because it is, well, interactive, but also because it shows how easy it is to generate an extension module with Shed Skin and use this in a larger program (a Pygame application in this case).
The biggest improvement (if you can call it this) for this release was to drop support for generic types. This simply means that functions that can be called with different kinds of types of objects for a single argument are no longer supported (unless these types have a common base-class, of course). Similarly, generic datastructures are no longer supported. Dropping support for generic types made the compiler core a lot simpler (and 10% smaller!), and removed the need for the -i command-line option.
Seeing the compiler core shrink so much, I was inspired to further refactor several messy areas of the compiler for readability (most notably in infer.py). This means the compiler core has also become a lot more readable/hackable with this release. I hope to continue this work of refactoring (and adding docstrings) with each release from now on, and invite anyone to help out here.
In general, I would really like to receive more help. For example, if someone could take over maintainership of the Windows version (and possible upgrade the MingW distribution that is packaged with it), that would be great. A suggestion on the coding side might be to add keyword support to generated extension modules (see extmod.py), or to optimize some builtins (for example, string/list slicing, see lib/builtin.?pp). But I would also be happy to receive more interesting test cases (most come from a single person at the moment) and/or more bug reports (which I get far too few of.)
Monday, March 23, 2009
Minilight Compiled
(Shed Skin is an experimental (restricted-)Python-to-C++ compiler.)
Minilight is an elegant minimal global illumination renderer, or raytracer, that uses triangle primitives and an octree spatial index. The original version consists of about 1,000 lines of C++, but there are several translations available at the homepage, such as ~500 line OCaml and Python versions. The Python version apparently clocks in 173 times later than the C++ version, while the OCaml version is only about 3 times slower.. Wouldn't it be interesting if we could make the Python version run faster than the OCaml version?
After some small changes (splitting up some dynamic variables, replacing calls to 'map', 'type'..), and applying some minor fixes to Shedskin, I was able to compile the Python version to C++ using Shedskin SVN. For the basic Cornell box example, it runs about 35 times faster than the modified Python version (which should again be a bit faster than the original Python version). It's probably still somewhat slower than the OCaml version, but not that much. To try to compile it yourself, you'll probably want to have more than 512 MB of RAM, and use the 'shedskin -i' option. The modified version can be found in /examples/minilight in SVN.
I'm guessing that the Python version can be made a bit faster in itself. What typically happens then is that the C++ version will get relatively even faster, because certain bottlenecks that don't really slow down Python will slow down C++ (such as allocating lots of objects). If you'd like to try and help beat OCaml, I'd be very interested in hearing about potential improvements (where readability doesn't suffer too much, obviously.)
Another program that I managed to compile, while much smaller, is one that solves a well-known chess puzzle. To compile knight2.py, I had to replace the 'key' argument to 'sorted' with a 'cmp' version (as 'key' is not yet supported), and fix a small bug in Shedskin ("'%*d' % (2, 7)" and such did not work yet.)
I'm always interested in hearing about other interesting test cases for Shedskin, as they seem hard to come by. Please also consider sending in bug reports. Both types of feedback are essential for me to keep working on Shedskin!
Update: After some tweaking, it is now more than 60 times faster than the original here. See the comments for more information.
Update 2: I removed the -i option with Shedskin 0.1.1, so it is not necessary anymore.
Minilight is an elegant minimal global illumination renderer, or raytracer, that uses triangle primitives and an octree spatial index. The original version consists of about 1,000 lines of C++, but there are several translations available at the homepage, such as ~500 line OCaml and Python versions. The Python version apparently clocks in 173 times later than the C++ version, while the OCaml version is only about 3 times slower.. Wouldn't it be interesting if we could make the Python version run faster than the OCaml version?
After some small changes (splitting up some dynamic variables, replacing calls to 'map', 'type'..), and applying some minor fixes to Shedskin, I was able to compile the Python version to C++ using Shedskin SVN. For the basic Cornell box example, it runs about 35 times faster than the modified Python version (which should again be a bit faster than the original Python version). It's probably still somewhat slower than the OCaml version, but not that much. To try to compile it yourself, you'll probably want to have more than 512 MB of RAM, and use the 'shedskin -i' option. The modified version can be found in /examples/minilight in SVN.
I'm guessing that the Python version can be made a bit faster in itself. What typically happens then is that the C++ version will get relatively even faster, because certain bottlenecks that don't really slow down Python will slow down C++ (such as allocating lots of objects). If you'd like to try and help beat OCaml, I'd be very interested in hearing about potential improvements (where readability doesn't suffer too much, obviously.)
Another program that I managed to compile, while much smaller, is one that solves a well-known chess puzzle. To compile knight2.py, I had to replace the 'key' argument to 'sorted' with a 'cmp' version (as 'key' is not yet supported), and fix a small bug in Shedskin ("'%*d' % (2, 7)" and such did not work yet.)
I'm always interested in hearing about other interesting test cases for Shedskin, as they seem hard to come by. Please also consider sending in bug reports. Both types of feedback are essential for me to keep working on Shedskin!
Update: After some tweaking, it is now more than 60 times faster than the original here. See the comments for more information.
Update 2: I removed the -i option with Shedskin 0.1.1, so it is not necessary anymore.
Sunday, January 25, 2009
Shed Skin 0.1
(Shed Skin is an experimental (restricted-)Python-to-C++ compiler.)
After several years of development and thirty releases, I have finally released Shed Skin 0.1. While still experimental, all the basic pieces are now in place:
-type inference that works for many smallish programs
-support for a useful subset of Python
-support for a good subset of the standard libraries (about 17 modules, such as random, re, os, os.path..)
-extension module generation (including extension classes)
-support for the most common platforms
For this release, I was able to add a Jpeg decoder, at 1,200 lines, an iPod Shuffle programmer, at 600 lines, and a mastermind program to the set of example programs. This brings the total number of example programs to 30, at about 9,000 lines in total.
For Shed Skin 0.2, I hope to be able to focus on improving type inference scalability. So far, I've spent surprisingly little time on type inference itself, because what I had worked well enough for most test cases. I have quite a few ideas to further improve scalability, and am sure some of these will help a lot.
This release would not have been possible without the generous help of Google (through their GSOC and GHOP programs) and the following people (please let me know if I missed someone):
-B********e
-Brian Blais
-Paul Boddie
-Djamel Cherif
-Mark Dewing
-James Coughlan
-Michael Elkins
-FFAO
-Luis M. Gonzales
-Karel Heyse
-Denis de Leeuw Duarte
-Van Lindberg
-David Marek
-Jeff Miller
-Joaquin Abian Monux
-Harri Pasanen
-SirNotAppearingInThisTutorial
-Dave Tweed
-Jaroslaw Tworek
-Pavel Vinogradov
Finally, I dedicate this release to Hai Fang Ni. Thanks for all your support, and a happy Chinese newyear!!
After several years of development and thirty releases, I have finally released Shed Skin 0.1. While still experimental, all the basic pieces are now in place:
-type inference that works for many smallish programs
-support for a useful subset of Python
-support for a good subset of the standard libraries (about 17 modules, such as random, re, os, os.path..)
-extension module generation (including extension classes)
-support for the most common platforms
For this release, I was able to add a Jpeg decoder, at 1,200 lines, an iPod Shuffle programmer, at 600 lines, and a mastermind program to the set of example programs. This brings the total number of example programs to 30, at about 9,000 lines in total.
For Shed Skin 0.2, I hope to be able to focus on improving type inference scalability. So far, I've spent surprisingly little time on type inference itself, because what I had worked well enough for most test cases. I have quite a few ideas to further improve scalability, and am sure some of these will help a lot.
This release would not have been possible without the generous help of Google (through their GSOC and GHOP programs) and the following people (please let me know if I missed someone):
-B********e
-Brian Blais
-Paul Boddie
-Djamel Cherif
-Mark Dewing
-James Coughlan
-Michael Elkins
-FFAO
-Luis M. Gonzales
-Karel Heyse
-Denis de Leeuw Duarte
-Van Lindberg
-David Marek
-Jeff Miller
-Joaquin Abian Monux
-Harri Pasanen
-SirNotAppearingInThisTutorial
-Dave Tweed
-Jaroslaw Tworek
-Pavel Vinogradov
Finally, I dedicate this release to Hai Fang Ni. Thanks for all your support, and a happy Chinese newyear!!
Sunday, December 28, 2008
Some nice additions to /examples
(Shed Skin is an experimental (restricted-)Python-to-C++ compiler.)
Because the set of example programs available at the Shed Skin homepage (ssprogs.tgz) hasn't changed in a while, I decided to add some nice programs to it. With some relatively simple (but as usual, important) fixes, I was able to add support for a Jpeg decompressor (!), at 1200 lines, an iPod shuffle programmer, at 600 lines, and a multi-module mastermind program, at 300 lines.
I will be adding these programs to ssprogs.tgz with 0.0.31 (or 0.1), but for now they can be found here. They should compile using Shed Skin SVN (see the note in the top of rdb.py; TonyJpegDecoder.py should be run with two arguments, e.g. "python TonyJpegDecoder.py tiger1.jpg tiger1.bmp").
Thanks to ********** for sending me the jpeg decompressor, and the authors of these programs for the nice test cases!
I would have probably added more programs (and fixed more things) if I could find more interesting test cases, but these seem hard to come by. If you happen to know of any nice programs between 500 and, say, 2,000 lines that might be statically compiled, possibly after some minor changes, please let me know. Some more bug reports would also be very welcome.
Because the set of example programs available at the Shed Skin homepage (ssprogs.tgz) hasn't changed in a while, I decided to add some nice programs to it. With some relatively simple (but as usual, important) fixes, I was able to add support for a Jpeg decompressor (!), at 1200 lines, an iPod shuffle programmer, at 600 lines, and a multi-module mastermind program, at 300 lines.
I will be adding these programs to ssprogs.tgz with 0.0.31 (or 0.1), but for now they can be found here. They should compile using Shed Skin SVN (see the note in the top of rdb.py; TonyJpegDecoder.py should be run with two arguments, e.g. "python TonyJpegDecoder.py tiger1.jpg tiger1.bmp").
Thanks to ********** for sending me the jpeg decompressor, and the authors of these programs for the nice test cases!
I would have probably added more programs (and fixed more things) if I could find more interesting test cases, but these seem hard to come by. If you happen to know of any nice programs between 500 and, say, 2,000 lines that might be statically compiled, possibly after some minor changes, please let me know. Some more bug reports would also be very welcome.
Tuesday, December 02, 2008
Shed Skin 0.0.30
I have just released version 0.0.30 of Shed Skin, an experimental (restricted) Python-to-C++ compiler.
Most importantly, this release adds (efficient) support for user-defined classes in generated extension modules, which should make it much easier to integrate compiled code within larger projects. More specifically, compiled classes can now be instantiated on the CPython side, and instances can be passed freely between CPython and Shed Skin without any conversion taking place. (Instances of builtin classes are still (recursively) copied, though, at the moment..)
Another major improvement was contributed by FFAO: a new 'set' implementation, directly based on the CPython code. While I haven't tested it on many benchmarks, it is clear that is now much faster, and on one benchmark it even outperforms CPython on my system by about 35%.
Other notable changes include complex number support, mapping None to NULL instead of 0 and printing it as 'None', as well as an important type inference fix.
With support for user-defined classes in extension modules, it looks like all the major pieces are now there to do a 0.1 release. The only thing I'd really like to do before that, is to improve support for the 'os' module. Please let me know if you'd like to help out here! Hopefully, with many details out of the way, I can have another good look at type inference for 0.2..
Most importantly, this release adds (efficient) support for user-defined classes in generated extension modules, which should make it much easier to integrate compiled code within larger projects. More specifically, compiled classes can now be instantiated on the CPython side, and instances can be passed freely between CPython and Shed Skin without any conversion taking place. (Instances of builtin classes are still (recursively) copied, though, at the moment..)
Another major improvement was contributed by FFAO: a new 'set' implementation, directly based on the CPython code. While I haven't tested it on many benchmarks, it is clear that is now much faster, and on one benchmark it even outperforms CPython on my system by about 35%.
Other notable changes include complex number support, mapping None to NULL instead of 0 and printing it as 'None', as well as an important type inference fix.
With support for user-defined classes in extension modules, it looks like all the major pieces are now there to do a 0.1 release. The only thing I'd really like to do before that, is to improve support for the 'os' module. Please let me know if you'd like to help out here! Hopefully, with many details out of the way, I can have another good look at type inference for 0.2..
Friday, November 21, 2008
Shed Skin in the News
(Shed Skin is an experimental (restricted-)Python-to-C++ compiler).
Because a new version (0.0.30) is still some weeks away, I decided to post, in the meantime, some links to other people's blog entries about Shed Skin.
making-python-math-196-faster-with-shedskin
taking-on-shed-skin
more performance python
compiled kamaelia
Please let me know if I have missed an interesting entry.
And if you are really bored, here are also some links to (old) discussions on slashdot and osnews:
slashdot article
osnews article
Because a new version (0.0.30) is still some weeks away, I decided to post, in the meantime, some links to other people's blog entries about Shed Skin.
making-python-math-196-faster-with-shedskin
taking-on-shed-skin
more performance python
compiled kamaelia
Please let me know if I have missed an interesting entry.
And if you are really bored, here are also some links to (old) discussions on slashdot and osnews:
slashdot article
osnews article
Saturday, September 20, 2008
Shed Skin 0.0.29
I have just released version 0.0.29 of Shed Skin, an experimental (restricted) Python-to-C++ compiler. It's been a while since the last release (well, just under 4 months), because of work and vacation, but mostly because there have been so many changes.
Thanks to the work of Karel Heyse, Pavel Vinogradov, FFAO and David Marek, there is now a pretty much complete implementation of the datetime module. Thanks to a suggestion by Albert Hofkamp, I added support for most of the ConfigParser module (by compiling it with Shed Skin of course), which led me to fix several outstanding but important problems. For example, support for inheritance hierarchies was greatly improved, and mapping keys should now work, too ('%(key)x..' % some_dict).
Since the previous release, I've also gone through the pains of installing FreeBSD, OpenSolaris and 64-bit Ubuntu on a spare PC and testing Shed Skin on each of them, with the result that these platforms are now 'officially' supported. Please see the updated 'installation' section of the tutorial. I also received some GCC 4.3 patches (Seo Sanghyeon and Winterknight), which should work fine now, too. So I guess most platforms should be covered now.
Some other improvements:
-improved support for importing from nested modules such as os.path
-__init__ methods are much less a special-case
-improved support for calling ancestor methods (e.g. Parent.__init__)
-all example programs (ss-progs) now compile as extension modules
-staticmethod and property decorator support (Seo Sanghyeon)
-Shed Skin doesn't crash on highly dynamic/recursive types anymore, making it easier to debug programs to get them to compile
-several fixes in the re module, e.g. re.sub now accepts a replacement function
-tuple hash caching was disabled, as CPython doesn't do this either
For the full changelog, see the release notes wiki at the Googlecode site. Please try out the new release, and let me know about any problems. Note that I probably won't be very responsive for at least a week or so - it has been a somewhat difficult release, and I could use some rest.. :)
Thanks to the work of Karel Heyse, Pavel Vinogradov, FFAO and David Marek, there is now a pretty much complete implementation of the datetime module. Thanks to a suggestion by Albert Hofkamp, I added support for most of the ConfigParser module (by compiling it with Shed Skin of course), which led me to fix several outstanding but important problems. For example, support for inheritance hierarchies was greatly improved, and mapping keys should now work, too ('%(key)x..' % some_dict).
Since the previous release, I've also gone through the pains of installing FreeBSD, OpenSolaris and 64-bit Ubuntu on a spare PC and testing Shed Skin on each of them, with the result that these platforms are now 'officially' supported. Please see the updated 'installation' section of the tutorial. I also received some GCC 4.3 patches (Seo Sanghyeon and Winterknight), which should work fine now, too. So I guess most platforms should be covered now.
Some other improvements:
-improved support for importing from nested modules such as os.path
-__init__ methods are much less a special-case
-improved support for calling ancestor methods (e.g. Parent.__init__)
-all example programs (ss-progs) now compile as extension modules
-staticmethod and property decorator support (Seo Sanghyeon)
-Shed Skin doesn't crash on highly dynamic/recursive types anymore, making it easier to debug programs to get them to compile
-several fixes in the re module, e.g. re.sub now accepts a replacement function
-tuple hash caching was disabled, as CPython doesn't do this either
For the full changelog, see the release notes wiki at the Googlecode site. Please try out the new release, and let me know about any problems. Note that I probably won't be very responsive for at least a week or so - it has been a somewhat difficult release, and I could use some rest.. :)
Tuesday, June 03, 2008
Shed Skin 0.0.28
I have just released Shed Skin 0.0.28, with the following changes. Thanks to those mentioned for helping out!
- basic 'socket' support (Michael Elkins)
- support for os.{popen3, popen4} under UNIX (Jaroslaw Tworek)
- support for time.strptime under Windows (David Marek)
- options for changing output dir, disabling annotation (Dave Tweed)
- support for 'cmp' and 'reverse' arguments of 'sorted' and 'list.sort'
- fixes for cross-module default arguments
- important fixes for type inference and inheritance
- restore compatibility with Python 2.3
- many minor bugfixes
I would really like to receive more bug reports. Please try out the new version, and let me know about any problems.
With the socket support, 15 common modules are now largely supported. For a 0.1 release, I'd really like to have support for one more module: datetime. Thanks to the GHOP, there is a type model already (lib/datetime.py), so we only still need a C++ implementation..
- basic 'socket' support (Michael Elkins)
- support for os.{popen3, popen4} under UNIX (Jaroslaw Tworek)
- support for time.strptime under Windows (David Marek)
- options for changing output dir, disabling annotation (Dave Tweed)
- support for 'cmp' and 'reverse' arguments of 'sorted' and 'list.sort'
- fixes for cross-module default arguments
- important fixes for type inference and inheritance
- restore compatibility with Python 2.3
- many minor bugfixes
I would really like to receive more bug reports. Please try out the new version, and let me know about any problems.
With the socket support, 15 common modules are now largely supported. For a 0.1 release, I'd really like to have support for one more module: datetime. Thanks to the GHOP, there is a type model already (lib/datetime.py), so we only still need a C++ implementation..
Saturday, February 23, 2008
Shed Skin 0.0.27
Thanks to the GHOP students, this new release contains support for 're' (via libpcre!), 'time' (except strptime under windows) and 'fnmatch' and 'glob', as well as much improved support for 'os' (POSIX), and also a typemodel for 'datetime'. This means Shed Skin now supports (most of) the following standard library modules:
bisect, collections, copy, fnmatch, getopt, glob, math, os, os.path, random, re, string, sys and time.
If anyone is interested in implementing/bootstrapping 'datetime', 'socket' or in improving 'os' support (esp. under Windows) or other modules, please let me know!
I got more help for this release. Brian Blais helped fix OSX support again, and added support for building extension modules under OSX. And Mark Dewing (in attempts to compile parts of Quameon, a monte carlo quantum atom simulator) sent in several bug reports. Several other people also sent in bug reports.
Of course I have added some things myself, too ;) For one, I added basic support for the 'staticmethod' and 'property' builtins. I also (finally, might I say) split up the compiler core (ss.py) into several files, added many fixes for multi-dir/multi-file projects, further optimized a few builtins, and fixed many minor issues.
In all, I'm very happy with this release, and hope to get as much help for the next release!
bisect, collections, copy, fnmatch, getopt, glob, math, os, os.path, random, re, string, sys and time.
If anyone is interested in implementing/bootstrapping 'datetime', 'socket' or in improving 'os' support (esp. under Windows) or other modules, please let me know!
I got more help for this release. Brian Blais helped fix OSX support again, and added support for building extension modules under OSX. And Mark Dewing (in attempts to compile parts of Quameon, a monte carlo quantum atom simulator) sent in several bug reports. Several other people also sent in bug reports.
Of course I have added some things myself, too ;) For one, I added basic support for the 'staticmethod' and 'property' builtins. I also (finally, might I say) split up the compiler core (ss.py) into several files, added many fixes for multi-dir/multi-file projects, further optimized a few builtins, and fixed many minor issues.
In all, I'm very happy with this release, and hope to get as much help for the next release!
Wednesday, January 16, 2008
Shed Skin 0.0.26
After my 'rant' about not getting much help with Shed Skin, I received a lot of feedback on how to improve packaging and usability in general. Several people even started helping out with this :-) Most notably, Paul helped me to create a Debian package, and James wrote a tutorial, which has by now replaced the README. Thanks guys!!
Shortly after my rant, somehow I also got involved in the Google GHOP project, in which high school students are paid to help out with open source projects. Some of these students turned out to be a real help, and one of them even managed to add complete 're' support to Shed Skin, using PCRE (perl-compatible-regular-expression library, thanks Cyril for the suggestion; this will be available in the next release!).
All of this, in turn, inspired me to add (almost) complete support for os.path and collections.defaultdict (deque was already supported). The combined result of all this should be a much more 'usable' project, although many things can still be improved of course. For the next release, I am planning on splitting up the compiler core (ss.py), and adding support for time, datetime, socket and as mentioned re.
When all this works well, I think the time has finally come to have another look at type inference scalability. Two ideas that stand out in my mind are iterative deepening (restart after an increasing number of iterations, each time combining everything that was learned) and selector-based filtering (basically, the idea is that for a method call x.bleh(), obviously x can only be of a type that has a bleh method, though it can be generalized further, e.g. for x=y).
Anyway, getting back to the subject, I have just released Shed Skin 0.0.26. It took me a while to get there, but this is mostly because it has many improvements:
-support for:
-most os.path methods (bootstrapped using Shed Skin)
-many os methods (ghop; many still remaining)
-collections.defaultdict (completing collections)
-getopt.gnu_getopt (bootstrapped, completing getopt)
-5 of the last 7 missing str methods (ghop)
-optimized string addition (a+b+c..)
-improved locale support (ghop)
-removed many leading underscores from generated code
-new documentation/tutorial
-added a Debian package
-many minor bugfixes
Additionally, I moved the project from SourceForge (sorry, guys) to Google code hosting. This is the new address:
http://shedskin.googlecode.com
Shortly after my rant, somehow I also got involved in the Google GHOP project, in which high school students are paid to help out with open source projects. Some of these students turned out to be a real help, and one of them even managed to add complete 're' support to Shed Skin, using PCRE (perl-compatible-regular-expression library, thanks Cyril for the suggestion; this will be available in the next release!).
All of this, in turn, inspired me to add (almost) complete support for os.path and collections.defaultdict (deque was already supported). The combined result of all this should be a much more 'usable' project, although many things can still be improved of course. For the next release, I am planning on splitting up the compiler core (ss.py), and adding support for time, datetime, socket and as mentioned re.
When all this works well, I think the time has finally come to have another look at type inference scalability. Two ideas that stand out in my mind are iterative deepening (restart after an increasing number of iterations, each time combining everything that was learned) and selector-based filtering (basically, the idea is that for a method call x.bleh(), obviously x can only be of a type that has a bleh method, though it can be generalized further, e.g. for x=y).
Anyway, getting back to the subject, I have just released Shed Skin 0.0.26. It took me a while to get there, but this is mostly because it has many improvements:
-support for:
-most os.path methods (bootstrapped using Shed Skin)
-many os methods (ghop; many still remaining)
-collections.defaultdict (completing collections)
-getopt.gnu_getopt (bootstrapped, completing getopt)
-5 of the last 7 missing str methods (ghop)
-optimized string addition (a+b+c..)
-improved locale support (ghop)
-removed many leading underscores from generated code
-new documentation/tutorial
-added a Debian package
-many minor bugfixes
Additionally, I moved the project from SourceForge (sorry, guys) to Google code hosting. This is the new address:
http://shedskin.googlecode.com
Saturday, November 24, 2007
Shed Skin: Call for participation
Okay, so I have built this pretty cool (restricted) Python-to-C++ compiler, that actually works for many not-too-large programs. It currently builds (simple) extension modules, and shows massive speedups for many programs, often outperforming Psyco by a factor. As I recently showed on my blog, it also integrates nicely with Parallel Python. I would have thought the latter would have given rise to at least one comment..
Am I the only one seeing the potential of an implicitly statically typed Python-like-language that runs at practically the same speed as C++? Is it that being ahead of the time makes you completely misunderstood, even by pretty smart hackers? I remember developing something quite like Wikipedia many years ago, and being unable to convince anyone of why this would be a good idea.. Of course that doesn't mean my compiler is a good idea (I wish), but I'm sure not going to give up as easily this time.
So, once again, I'd like to ask for more participation. Many open source users don't realize, I think, how open source projects, especially new ones, thrive on user feedback. Programming is debugging, and there's nothing more satisfying than fixing particular problems users encounter. Sure, I could find most bugs myself, but I don't want to end up in a nut house, and at some point there has to be some kind of community process. Patches are very useful too, even simple ones, as they often trigger more patches by me. Extension module support, for example, started out as a simple proof-of-concept patch sent in by a user.
I'd very much like to take Shed Skin further forward, but I need your help to do so! Please visit the homepage and send in bug reports or join the mailing list and start some discussions. I can also be hired to work on specific features of course :) So hurry to the homepage:
http://mark.dufour.googlepages.com
Am I the only one seeing the potential of an implicitly statically typed Python-like-language that runs at practically the same speed as C++? Is it that being ahead of the time makes you completely misunderstood, even by pretty smart hackers? I remember developing something quite like Wikipedia many years ago, and being unable to convince anyone of why this would be a good idea.. Of course that doesn't mean my compiler is a good idea (I wish), but I'm sure not going to give up as easily this time.
So, once again, I'd like to ask for more participation. Many open source users don't realize, I think, how open source projects, especially new ones, thrive on user feedback. Programming is debugging, and there's nothing more satisfying than fixing particular problems users encounter. Sure, I could find most bugs myself, but I don't want to end up in a nut house, and at some point there has to be some kind of community process. Patches are very useful too, even simple ones, as they often trigger more patches by me. Extension module support, for example, started out as a simple proof-of-concept patch sent in by a user.
I'd very much like to take Shed Skin further forward, but I need your help to do so! Please visit the homepage and send in bug reports or join the mailing list and start some discussions. I can also be hired to work on specific features of course :) So hurry to the homepage:
http://mark.dufour.googlepages.com
Thursday, October 18, 2007
Shed Skin and Parallel Python
Shed Skin is an experimental Python to C++ compiler. Parallel Python allows for clean and simple parallelization of Python processes over multiple cores. Wouldn't it be cool if we could combine the two? Of course I wouldn't be writing this if I hadn't tried. Here's how to do it.
Create an extension module with Shed Skin, containing function(s) you'd like to use in parallel. For example, we might use the partial sum function from the Parallel Python website:
def part_sum(start, end):
..
return sum
For Shed Skin's type inference to work, part_sum must be called from somewhere:
if __name__ == '__main__':
part_sum(1, 1)
Creating an extension module is simple (suppose the module is named meuk.py):
ss -e meuk
make
Because Parallel Python expects pure-Python code, we must call our compiled function via a pure-Python wrapper:
def part_sum(start, end):
import meuk
return meuk.part_sum(start, end)
In order for Parallel Python to find our extension module (at least on my Ubuntu system), we must issue this in advance:
export PYTHONPATH=$pwd
And there you have it. Here are some timings:
no extension module, 1 worker: 11.3 seconds
no extension module, 2 workers: 6.2 seconds
extension module, 1 worker: 0.6 seconds
extension module, 2 workers: 0.3 seconds
Create an extension module with Shed Skin, containing function(s) you'd like to use in parallel. For example, we might use the partial sum function from the Parallel Python website:
def part_sum(start, end):
..
return sum
For Shed Skin's type inference to work, part_sum must be called from somewhere:
if __name__ == '__main__':
part_sum(1, 1)
Creating an extension module is simple (suppose the module is named meuk.py):
ss -e meuk
make
Because Parallel Python expects pure-Python code, we must call our compiled function via a pure-Python wrapper:
def part_sum(start, end):
import meuk
return meuk.part_sum(start, end)
In order for Parallel Python to find our extension module (at least on my Ubuntu system), we must issue this in advance:
export PYTHONPATH=$pwd
And there you have it. Here are some timings:
no extension module, 1 worker: 11.3 seconds
no extension module, 2 workers: 6.2 seconds
extension module, 1 worker: 0.6 seconds
extension module, 2 workers: 0.3 seconds
Tuesday, October 16, 2007
Shed Skin 0.0.24, 0.0.25
I've just released Shed Skin 0.0.25. Together with the (unannounced) 0.0.24 release, there have been some interesting changes. Most importantly perhaps, Shed Skin now caches (most) 1-length strings, which can improve performance dramatically for string-intensive programs. I also performed a long-overdue rewrite of the virtual function detection code, which should work much more reliably now, at least for relatively simple cases :)
0.0.24:
-1-length string caching
0.0.25
-improved detection of virtual functions
-further set optimizations
-fix for extension modules and certain default arguments
-exhaustive checking of C++ keywords
-fix for some combinations of arguments to min, max
-several minor bug fixes
As always, I could really use more help in pushing Shed Skin forward. Let me know if you'd like to help out, but are not sure where to begin.
0.0.24:
-1-length string caching
0.0.25
-improved detection of virtual functions
-further set optimizations
-fix for extension modules and certain default arguments
-exhaustive checking of C++ keywords
-fix for some combinations of arguments to min, max
-several minor bug fixes
As always, I could really use more help in pushing Shed Skin forward. Let me know if you'd like to help out, but are not sure where to begin.
Monday, August 20, 2007
Shed Skin 0.0.23
I have just released Shed Skin 0.0.23. It doesn't contain the type inference scalability improvements I was working on, but it does have quite a few bug fixes and minor feature additions. Here's a list of changes:
-support for __iadd__, __imul__ and such (except __ipow__ and __imod__)
-some overdue set optimizations
-fix for string formatting problem (%% did not always work)
-extension module stability fixes
-fix for particular inheritance problem
-other minor bugfixes, cleanups, and error messages
I could really use some systematic help in pushing Shedskin further. Some ideas:
-send in bug reports - these are extremely valuable and motivating to me, yet I don't receive many..
-find out why test 148 is currently broken under windows
-add datetime, re or socket support
-look into supporting custom classes in generated extension modules
-write a Shedskin tutorial for 'novice' programmers
-systemically test performance and suggest and work on improvements
-investigate replacements for std::string and __gnu_cxx::hash_set
-perform janitorial-type work in ss.py and lib/builtin.?pp
-support extension modules under OSX (OSX gives me accute mental RSI)
-add more tests to unit.py
-support for __iadd__, __imul__ and such (except __ipow__ and __imod__)
-some overdue set optimizations
-fix for string formatting problem (%% did not always work)
-extension module stability fixes
-fix for particular inheritance problem
-other minor bugfixes, cleanups, and error messages
I could really use some systematic help in pushing Shedskin further. Some ideas:
-send in bug reports - these are extremely valuable and motivating to me, yet I don't receive many..
-find out why test 148 is currently broken under windows
-add datetime, re or socket support
-look into supporting custom classes in generated extension modules
-write a Shedskin tutorial for 'novice' programmers
-systemically test performance and suggest and work on improvements
-investigate replacements for std::string and __gnu_cxx::hash_set
-perform janitorial-type work in ss.py and lib/builtin.?pp
-support extension modules under OSX (OSX gives me accute mental RSI)
-add more tests to unit.py
Friday, June 29, 2007
Shed Skin 0.0.22
after being bogged down with work for a few weeks, I got back to development again. the trigger was a proof-of-concept patch sent in by Harri Pasanen for generating extension modules. previously I was waiting for someone else to fully tackle this, but his patch was quite simple, and I found out it easily works under mingw, too. so I generalised things a bit, and released it as part of version 0.0.22. the process of building an extension module is now quite simply 'ss -e ..' and 'make' (note it doesn't work under OSX, yet).
there are some limitations though as to the way this works:
-only builtin scalar and containers can be passed/returned (int, float, str, list, tuple, dict, set)
-arguments/returned objects are completely copied/converted at call/return time (i.e., including contained objects)
-global variables are considered constant, and converted at module initialization time
consider this simple program, mod_name.py:
some_var = [1,2,3]
def some_func(x):
return 2*x
if __name__ == '__main__':
some_func(1) # obviously, this is needed for type inference to work
to compile this down to an extension module, simply use the new '-e' command-line parameter:
ss -e mod_name
make
that's it. now the program can be used from an arbitrary python program/prompt:
>>>import mod_name
>>>dir(mod_name)
>>>mod_name.some_var
>>>mod_name.some_func(1)
I'm hoping someone else can add support for custom classes and find out how to get this to work under OSX!
there are some limitations though as to the way this works:
-only builtin scalar and containers can be passed/returned (int, float, str, list, tuple, dict, set)
-arguments/returned objects are completely copied/converted at call/return time (i.e., including contained objects)
-global variables are considered constant, and converted at module initialization time
consider this simple program, mod_name.py:
some_var = [1,2,3]
def some_func(x):
return 2*x
if __name__ == '__main__':
some_func(1) # obviously, this is needed for type inference to work
to compile this down to an extension module, simply use the new '-e' command-line parameter:
ss -e mod_name
make
that's it. now the program can be used from an arbitrary python program/prompt:
>>>import mod_name
>>>dir(mod_name)
>>>mod_name.some_var
>>>mod_name.some_func(1)
I'm hoping someone else can add support for custom classes and find out how to get this to work under OSX!
Tuesday, March 27, 2007
Shed Skin 0.0.20 and 0.0.21
development has been continuing at a steady pace, resulting in two new releases. I haven't gotten around to cleaning up and improving tuple support, as of course I got bogged down into other details. the first release essentially combines many minor fixes:
Release Name: 0.0.20
-improvements to aug-assignments with subscripting ('a[x, y] += 1' and such)
-fixed some problems with slice-assignments ('a[1:-1] = [1,2])
-make integer division (/,//,divmod,floordiv) equal to CPython for negative/positive combinations of arguments
-make printing of floats closer to CPython
-move generic functions/methods to header file
-many small fixes (allow 'self' as function argument, list.extend takes iterable, __delitem__ overloading, raw_input(), 'return' from generator..)
-improved error checking for dynamic types
-optimize expr**2 and expr**3 using __power2 and __power3 functions
the next release is more interesting. first, a problem exhibited in a little program bearophile sent me (I wish there were two bearophiles :-)) gave me the insight that two things I was doing during type inference were more or less the same; this allowed me to generalize things and cut away about 100 lines. second, I added support for 'bisect', 'collections.deque' and improved 'copy' support. finally, there were some important dict optimizations (dict[..] += .. and dict.__getitem__ now only index once):
Release Name: 0.0.21
-important type inference fix/cleanup
-support for 'bisect', 'collections.deque' and 'string.maketrans'
-improved 'copy' support
-support for 'try, else' construction
-some optimizations ('dict[..] += ..', "''.join(sorted(str))")
-several minor bug fixes
Release Name: 0.0.20
-improvements to aug-assignments with subscripting ('a[x, y] += 1' and such)
-fixed some problems with slice-assignments ('a[1:-1] = [1,2])
-make integer division (/,//,divmod,floordiv) equal to CPython for negative/positive combinations of arguments
-make printing of floats closer to CPython
-move generic functions/methods to header file
-many small fixes (allow 'self' as function argument, list.extend takes iterable, __delitem__ overloading, raw_input(), 'return' from generator..)
-improved error checking for dynamic types
-optimize expr**2 and expr**3 using __power2 and __power3 functions
the next release is more interesting. first, a problem exhibited in a little program bearophile sent me (I wish there were two bearophiles :-)) gave me the insight that two things I was doing during type inference were more or less the same; this allowed me to generalize things and cut away about 100 lines. second, I added support for 'bisect', 'collections.deque' and improved 'copy' support. finally, there were some important dict optimizations (dict[..] += .. and dict.__getitem__ now only index once):
Release Name: 0.0.21
-important type inference fix/cleanup
-support for 'bisect', 'collections.deque' and 'string.maketrans'
-improved 'copy' support
-support for 'try, else' construction
-some optimizations ('dict[..] += ..', "''.join(sorted(str))")
-several minor bug fixes
Monday, February 12, 2007
Shed Skin 0.0.19
I have just released 0.0.19, with some interesting changes. Iterators and generators are supported now. Full random support was contributed by a user (thanks Jeff :-)), by converting a Python version to C++ using Shedskin. Interestingly, by default it uses the same random engine (Mersenne Twister) as CPython, so the two behave exactly the same. Jeff is now thinking about how to add fast matrix support, which would be really nice to have.. But there were also many small fixes in this release. Here is the sourceforge changelog:
-iterator and generator support (no generator expressions yet)
-full support for random module (converted from Python version)
-added itertools.{count, repeat, cycle}, dict.{iterkeys, itervalues, iteritems}
-added 3-argument integer version of __builtin__.pow
-logical and, or operations have correct return value now (e.g., '3 and 4')
-lots of minor bug fixes (improved % operator, added math.pow..)
The main focus for 0.0.20 will probably be to add support for tuples, with differently typed elements, of length greater than 2, up to some arbitrary length..
-iterator and generator support (no generator expressions yet)
-full support for random module (converted from Python version)
-added itertools.{count, repeat, cycle}, dict.{iterkeys, itervalues, iteritems}
-added 3-argument integer version of __builtin__.pow
-logical and, or operations have correct return value now (e.g., '3 and 4')
-lots of minor bug fixes (improved % operator, added math.pow..)
The main focus for 0.0.20 will probably be to add support for tuples, with differently typed elements, of length greater than 2, up to some arbitrary length..
Subscribe to:
Posts (Atom)