CPython 3.15 (JIT) Performance Showdown

Now that CPython 3.15 has been released, I decided to do a historical performance comparison similar in spirit to Sixteen Years of Python Performance. Obviously I wanted to include CPython 3.15, but I also wanted to compare CPython to other Python implementations/optimizers (as I'm the main author of Shed Skin), to get a feel for the relative performance.

The comparison was done for eight examples coming from TurboPython, an interesting up-and-coming restricted-Python transpiler. Caveats: these programs were originally ported from the Shed Skin examples. I also didn't include the latest PyPy, as 7.3.20 is the one on my system. In all, you can assume these results are slightly biased towards Shed Skin (so no need to complain about that in the comments! ;)).

Without further ado, here are the final numbers (geometric mean speedup over the eight programs, versus CPython 2.7):

So what do the numbers show? First of all, the 3.15 JIT is doing really well for these algorithmic-type programs, with a mean speedup going from 1.61x (without JIT) to 2.47x (with JIT)! For 3.14, the JIT barely makes a difference. This is probably the most important take-away. It's still some ways away from PyPy-level performance, but if the CPython JIT is finally taking off, it could quickly make similar large jumps over the next few releases, perhaps even closing the gap with PyPy substantially.

PyPy 7.3.20 gives a very nice mean speedup of 24.2x. Note that I gave both PyPy and CPython JIT some warm-up time, and only measured after that. This is quite an impressive speedup, given that PyPy has its hands tied behind its back, supporting all Python features!

As for the AOT transpilers, keep in mind that these operate on restricted-Python variants, so they do not work for arbitrary Python code. It takes some work to make your code compatible, and sometimes that is not an option, but if you put in the work you can expect a larger speedup than using CPython JIT/PyPy.

TurboPython does really well for a relatively young project, with a mean speedup of 41.1x. It is actually fastest for two of the eight programs. There are currently some performance issues for the 'sieve' program, but I'm sure those will be worked out with the next release.

Shed Skin itself has had many performance improvements over the past year, so I included 0.9.12 for reference. The reason for testing with Shed Skin main is that I didn't want to do another release just for this comparison (but I could have probably easily released it as 0.9.15).

The full results (with per-program measurements) can be found here.

Shed Skin v0.9.14, v1.0 coming soon!

I have just released version 0.9.14 of Shed Skin, a restricted-python to C++ transpiler. Shed Skin allows one to effectively convert (or transpile) pure Python code to highly optimized machine code. This comes at the cost though of having to conform to a seriously restricted subset of Python features/libraries. Although, it is possible to generate extension modules, that can be used in larger, unrestricted, programs.

While the previous release was just a month ago, I felt like there were so many important improvements already that I didn't want them to wait for the 1.0 release which looks like it may finally happen within a few months! From a completely rewritten and faster, more scalable, type inference engine, to full Unicode support, to almost complete feature parity with Python 3.15 (for the supported modules), to support for heterogeneous 3-len tuples - it has been a hectic month :)

With all this in place, I'm starting to plan the final pieces I would like to see in place for a 1.0 release. There is still quite a bit of work left, but much of it is mechanical and just working through a long list of minor decisions and details.. It should be possible to have all that done around the end of the year.

I'm also happy to mention a relatively new but similar project here, TurboPython, which may be of interest to those reading this. The most interesting difference with Shed Skin is that it uses a RUST-like memory model: "TurboPython is a general-purpose, statically typed language that compiles Python through C++ to a native binary. It combines Python's syntax with an ownership model that gives memory safety and predictable performance—no garbage collector, no automatic reference counting, no GIL."

Shed Skin restricted-Python-to-C++ compiler v0.9.13 released

I have just released version 0.9.13 of Shed Skin, a restricted-python to C++ transpiler. Shed Skin allows one to effectively convert (or transpile) pure Python code to highly optimized machine code. This comes at the cost though of having to conform to a seriously restricted subset of Python features/libraries. Programs currently also cannot be too large, although it is possible to generate extension modules, that can be used in larger programs.

The following screenshot is of a DOOM engine, that becomes more than 30 times faster after transpilation. The engine is compiled with Shed Skin, then imported in a larger program that uses Pygame for the UI (see also a before-after video).

Version 0.9.13 is the first version that was heavily improved by the use of AI (more specifically, Claude - thanks to Shakeeb for starting this). It was able to spot (and fix) many bugs, especially in the C++ backend, but also test gaps. It also helped to add support for many missing features in Python 3.15. We are now actually pretty close to full compatibility there, at least with regards to the supported modules and everything that is compatible with Shed Skin (no os.walk yet, for example, as it uses heterogenous tuples of length > 2, something we may fix for 0.9.14).

The use of AI should make it much easier to contribute as well. For example, if your program doesn't work, AI can minimize it and produce a useful bug report to submit to the project. It can even start to try and fix the problem, create new tests and so on. Or otherwise look for gaps/bugs to work on.