CPython 3.15 (JIT) Performance Showdown

Now that CPython 3.15 has been released, I decided to do a historical performance comparison similar in spirit to Sixteen Years of Python Performance. Obviously I wanted to include CPython 3.15, but I also wanted to compare CPython to other Python implementations/optimizers (as I'm the main author of Shed Skin), to get a feel for the relative performance.

The comparison was done for eight examples coming from TurboPython, an interesting up-and-coming restricted-Python transpiler. Caveats: these programs were originally ported from the Shed Skin examples. I also didn't include the latest PyPy, as 7.3.20 is the one on my system. In all, you can assume these results are slightly biased towards Shed Skin (so no need to complain about that in the comments! ;)).

Without further ado, here are the final numbers (geometric mean speedup over the eight programs, versus CPython 2.7):

So what do the numbers show? First of all, the 3.15 JIT is doing really well for these algorithmic-type programs, with a mean speedup going from 1.61x (without JIT) to 2.47x (with JIT)! For 3.14, the JIT barely makes a difference. This is probably the most important take-away. It's still some ways away from PyPy-level performance, but if the CPython JIT is finally taking off, it could quickly make similar large jumps over the next few releases, perhaps even closing the gap with PyPy substantially.

PyPy 7.3.20 gives a very nice mean speedup of 24.2x. Note that I gave both PyPy and CPython JIT some warm-up time, and only measured after that. This is quite an impressive speedup, given that PyPy has its hands tied behind its back, supporting all Python features!

As for the AOT transpilers, keep in mind that these operate on restricted-Python variants, so they do not work for arbitrary Python code. It takes some work to make your code compatible, and sometimes that is not an option, but if you put in the work you can expect a larger speedup than using CPython JIT/PyPy.

TurboPython does really well for a relatively young project, with a mean speedup of 41.1x. It is actually fastest for two of the eight programs. There are currently some performance issues for the 'sieve' program, but I'm sure those will be worked out with the next release.

Shed Skin itself has had many performance improvements over the past year, so I included 0.9.12 for reference. The reason for testing with Shed Skin main is that I didn't want to do another release just for this comparison (but I could have probably easily released it as 0.9.15).

The full results (with per-program measurements) can be found here.