Thank you! :)
In the `wasmi-benchmarks` suite we support ~20 different Wasm runtimes and compare their performance with each other, including optimizing JITs such as Wasmtime/Wasmer Cranelift and baseline JITs such as Wasmtime Winch and Wasmer Singlepass.
The geomean of performance of Wasmi compared to baseline JITs across all benchmarks in the repository ranges from 2.5-5.2x slower depending on hardware.
And compared to opimizing JITs geomean ranges from 5.3-10.7x slower.
Wasmtime's Pulley is a very interesting interpreter. It isn't the fastest but it is the only Wasm interpreter that sits behind an elaborate optimization pipeline. Thus if you feed unoptimized Wasm, it would likely outperform the other interpreters. However, unoptimized Wasm is extremely uncommon.
When I've measured this in the past I've seen Wasmtime's Cranelift compiler generating code that runs roughly 1.5x-1.9x slower than native (LLVM) on SPEC 2017 workloads, with x86-64 being closer to 1.5x and AArch64 closer to 1.9x. From what I recall, interpreter performance was generally more like 10x slower than native, with some workloads that involved heavy cryptography/SIMD being 30x slower.
No, they're not getting close. Top performing Wasm runtimes (like those in V8 and JSC) are generally between 10% (for pure math workloads) and 2x (conservatively, for ef allocation heavy ones) the speed of equivalent native implementations. But that's including the JIT compilation tiers; the performance of interpreters alone lags by an order of magnitude.
Surely you mean the "time spent," not the "speed" - as an interpreter would have an overhead, not magically speed up WASM execution. Somewhat related note, we need better tools for PGO within native compiled programs.
hey, I'm not super familliar with webassembly interpreters in general so id ask here:
What's the usecase? like I guess edge, chrome etc already have their own interpreters for webassembly built in. do you aim to replace those and be bundled with them?
Or is this for other browsers? or even just other apps (whats the usecase there as opposed to just native execution)
Wasmi does not directly compete with the large JIT engines such as Wasmtime and V8. Instead, Wasmi tries to fit perfectly into its niche. Its main purpose is that it is very easy to embed and provides great performance for those use-cases.
As detailed in the article, Wasmi is already used a lot for plugin systems, as game engine, as engine for executing smart contracts, and even as engine to run apps in experimental operating systems that have native Wasm support. It is also useful for cloud hosts that do not trust their inputs but need fast startup times and deterministic execution.
Furthermore, there are platforms such as iOS that outright forbid using JITs, so interpreters are the only option.
Fun fact: Wasm interpreter usually can even be embedded into Wasm environments themselves by compiling them to Wasm. Wasmi ran inside Wasmtime when it was used at Parity Technologies. This allowed them to hot-patch the Wasm runtime (Wasmi) without downtime.
> Wasmi is an efficient and feature-rich WebAssembly (Wasm) interpreter. It is an excellent choice for IoT devices, plugin systems (Typst, Zellij, Josh), cloud hosts, smart contracts (Soroban, Ripple) and even for your lightweight game consoles (Firefly Zero).
Hi there! It's really great seeing someone putting in the work to make a WASM interpreter fast :) I'm very much interested in this because I want my game's scripting system to simply run WASM so I need a fast WASM interpreter because iOS forbids JIT and most gaming consoles allegedly do so as well. I know it's NDA gated so one has to be light on details, but have you heard about people using wasmi interpreter on one of the major gaming consoles? I could imagine how that also makes awesome modding possible, especially when being able to limit how many resources those mods are allowed to consume.
I am not aware of any of the major gaming consoles uses Wasm or Wasmi in particular for their engine but I am sure they'd let us know if they ever used Wasm as execution model for all their games.
What's more likely is that Wasm is used in some indie games for those major game consoles.
I know of one game engine (Firefly-zero) and one game where Wasmi is used as game engine and plugin engine respectively. There likely are more, but that's what I know for a certain.
I experimented with running Doom using Wasmi and it even works in the browser, thus in a double sandbox where Wasmi itself is compiled to Wasm: (references in the article)
https://wasmi-labs.github.io/wasmi-doom/
If Wasmi supports all the Wasm proposals that you need for PGlite and if Wasmi supports all the WASI features you need (Wasmi only supported the standard WASI features without extensions), then Wasmi should work for your use-case. :)
I've been poking at a C version of this (https://github.com/dan-eicher/javelina) which would be interesting to benchmark against as it does a similar tail-calling dispatch mechanism. Plus copy-and-patch JIT but that probably only works on x86-64 as that's the only place I've ever tested it. The main differences from a brief skim of TFA is mine doesn't have any fallback (so non-tail calls will blow up the C stack) and the function calls always go through the trampoline so the VM doesn't have to care if it's calling JIT or interpreted code which, I'm assuming, your function pointer embedding thing is designed to optimize away.
And the JavaCard firewall algorithm would be an interesting non-spec addition to a wasm VM which is running code you really, really don't want to escape the sandbox. Something to look into for inspiration on the subject, perhaps? Not sure if there's any sort of proposal for sandboxing these things as I just took the spec file and implemented it using the dodgy weasels where it was mainly to see how far they've come with no real plan to use it for anything so kept it strictly to what the spec said a wasm interpreter needs to do.
Anyhoo, didn't really realize there were so many different projects doing the same thing, kind of interesting, actually...
If you think that your Wasm interpreter is stable and kinda production ready enough, you might want to file a PR to the wasmi-benchmarks repo to add support for your Wasm runtime.
Would certainly be another great addition to have it.
This would allow comparing your engine to all the others.
> and even for your lightweight game consoles (Firefly Zero).
On that note, does the "shape" of the benchmarks change on lower-powered hardware in any way? (Aside from the obvious change if memory overhead requires switching from direct to indirect threaded code)
From my experiences indirect-threading and direct-threading are only ~10-15% of performance apart.
However, the switch-loop dispatch that is used on platforms that do not support tail-calls can be a lot slower. However, the slowdown highly depends on the underlying hardware. For example, on Apple Silicon the slow-down is huge, whereas on Intel the slowdown isn't that drastic.
Unfortunately, I haven't tested any of the Wasm runtimes on low-powered hardware so far but that would be a great addition and I'd be extremely interested in how the fast interpreters such as Wasmi, Wasm3 and Stitch perform there. From what I know Wasm3 was optimized for those targets, so it might fare well and if Wasmi does not yet perform well there it should be fairly easy to catch up since the architectural foundation is similar.
Also, Wasmi's auto-dispatch feature that automatically detects if tail-calls can be used is very conservative. We might be able to cover more targets in the future with it, thus avoiding the slower switch-loop for more platforms eventually.
From the people that use Wasmi on lower-powered hardware (e.g. the Firefly-zero people) they seem to be very happy with Wasmi's performance so far.
Very cool!
How does the performance of this wasm interpreter compare to native execution? Are we getting close?
And you’re comparing against wasmtime.pulley, which is their optimising interpreter. How does it stack up against wasmtime’s cranelift compiler?
Thank you! :) In the `wasmi-benchmarks` suite we support ~20 different Wasm runtimes and compare their performance with each other, including optimizing JITs such as Wasmtime/Wasmer Cranelift and baseline JITs such as Wasmtime Winch and Wasmer Singlepass.
The geomean of performance of Wasmi compared to baseline JITs across all benchmarks in the repository ranges from 2.5-5.2x slower depending on hardware. And compared to opimizing JITs geomean ranges from 5.3-10.7x slower.
Wasmtime's Pulley is a very interesting interpreter. It isn't the fastest but it is the only Wasm interpreter that sits behind an elaborate optimization pipeline. Thus if you feed unoptimized Wasm, it would likely outperform the other interpreters. However, unoptimized Wasm is extremely uncommon.
When I've measured this in the past I've seen Wasmtime's Cranelift compiler generating code that runs roughly 1.5x-1.9x slower than native (LLVM) on SPEC 2017 workloads, with x86-64 being closer to 1.5x and AArch64 closer to 1.9x. From what I recall, interpreter performance was generally more like 10x slower than native, with some workloads that involved heavy cryptography/SIMD being 30x slower.
No, they're not getting close. Top performing Wasm runtimes (like those in V8 and JSC) are generally between 10% (for pure math workloads) and 2x (conservatively, for ef allocation heavy ones) the speed of equivalent native implementations. But that's including the JIT compilation tiers; the performance of interpreters alone lags by an order of magnitude.
Between 10% and 100% ? Am I understanding that right?
I put that poorly: read as 110% and 200%
Surely you mean the "time spent," not the "speed" - as an interpreter would have an overhead, not magically speed up WASM execution. Somewhat related note, we need better tools for PGO within native compiled programs.
hi, author here, ready to answer all your questions! (sorry for the delay, 2nd-chance post)
hey, I'm not super familliar with webassembly interpreters in general so id ask here:
What's the usecase? like I guess edge, chrome etc already have their own interpreters for webassembly built in. do you aim to replace those and be bundled with them?
Or is this for other browsers? or even just other apps (whats the usecase there as opposed to just native execution)
Wasmi does not directly compete with the large JIT engines such as Wasmtime and V8. Instead, Wasmi tries to fit perfectly into its niche. Its main purpose is that it is very easy to embed and provides great performance for those use-cases.
As detailed in the article, Wasmi is already used a lot for plugin systems, as game engine, as engine for executing smart contracts, and even as engine to run apps in experimental operating systems that have native Wasm support. It is also useful for cloud hosts that do not trust their inputs but need fast startup times and deterministic execution.
Furthermore, there are platforms such as iOS that outright forbid using JITs, so interpreters are the only option.
Fun fact: Wasm interpreter usually can even be embedded into Wasm environments themselves by compiling them to Wasm. Wasmi ran inside Wasmtime when it was used at Parity Technologies. This allowed them to hot-patch the Wasm runtime (Wasmi) without downtime.
TFA's third paragraph:
> Wasmi is an efficient and feature-rich WebAssembly (Wasm) interpreter. It is an excellent choice for IoT devices, plugin systems (Typst, Zellij, Josh), cloud hosts, smart contracts (Soroban, Ripple) and even for your lightweight game consoles (Firefly Zero).
Hi there! It's really great seeing someone putting in the work to make a WASM interpreter fast :) I'm very much interested in this because I want my game's scripting system to simply run WASM so I need a fast WASM interpreter because iOS forbids JIT and most gaming consoles allegedly do so as well. I know it's NDA gated so one has to be light on details, but have you heard about people using wasmi interpreter on one of the major gaming consoles? I could imagine how that also makes awesome modding possible, especially when being able to limit how many resources those mods are allowed to consume.
I am not aware of any of the major gaming consoles uses Wasm or Wasmi in particular for their engine but I am sure they'd let us know if they ever used Wasm as execution model for all their games.
What's more likely is that Wasm is used in some indie games for those major game consoles.
I know of one game engine (Firefly-zero) and one game where Wasmi is used as game engine and plugin engine respectively. There likely are more, but that's what I know for a certain.
I experimented with running Doom using Wasmi and it even works in the browser, thus in a double sandbox where Wasmi itself is compiled to Wasm: (references in the article) https://wasmi-labs.github.io/wasmi-doom/
Can I use it to embed PGlite in an app and distribute it like that? ie not to dependend on the host JIT/interpreter
If Wasmi supports all the Wasm proposals that you need for PGlite and if Wasmi supports all the WASI features you need (Wasmi only supported the standard WASI features without extensions), then Wasmi should work for your use-case. :)
Wasmi itself can be compiled to WebAssembly.
I've been poking at a C version of this (https://github.com/dan-eicher/javelina) which would be interesting to benchmark against as it does a similar tail-calling dispatch mechanism. Plus copy-and-patch JIT but that probably only works on x86-64 as that's the only place I've ever tested it. The main differences from a brief skim of TFA is mine doesn't have any fallback (so non-tail calls will blow up the C stack) and the function calls always go through the trampoline so the VM doesn't have to care if it's calling JIT or interpreted code which, I'm assuming, your function pointer embedding thing is designed to optimize away.
And the JavaCard firewall algorithm would be an interesting non-spec addition to a wasm VM which is running code you really, really don't want to escape the sandbox. Something to look into for inspiration on the subject, perhaps? Not sure if there's any sort of proposal for sandboxing these things as I just took the spec file and implemented it using the dodgy weasels where it was mainly to see how far they've come with no real plan to use it for anything so kept it strictly to what the spec said a wasm interpreter needs to do.
Anyhoo, didn't really realize there were so many different projects doing the same thing, kind of interesting, actually...
Cool project!
If you think that your Wasm interpreter is stable and kinda production ready enough, you might want to file a PR to the wasmi-benchmarks repo to add support for your Wasm runtime.
Would certainly be another great addition to have it. This would allow comparing your engine to all the others.
> and even for your lightweight game consoles (Firefly Zero).
On that note, does the "shape" of the benchmarks change on lower-powered hardware in any way? (Aside from the obvious change if memory overhead requires switching from direct to indirect threaded code)
From my experiences indirect-threading and direct-threading are only ~10-15% of performance apart. However, the switch-loop dispatch that is used on platforms that do not support tail-calls can be a lot slower. However, the slowdown highly depends on the underlying hardware. For example, on Apple Silicon the slow-down is huge, whereas on Intel the slowdown isn't that drastic.
Unfortunately, I haven't tested any of the Wasm runtimes on low-powered hardware so far but that would be a great addition and I'd be extremely interested in how the fast interpreters such as Wasmi, Wasm3 and Stitch perform there. From what I know Wasm3 was optimized for those targets, so it might fare well and if Wasmi does not yet perform well there it should be fairly easy to catch up since the architectural foundation is similar.
Also, Wasmi's auto-dispatch feature that automatically detects if tail-calls can be used is very conservative. We might be able to cover more targets in the future with it, thus avoiding the slower switch-loop for more platforms eventually.
From the people that use Wasmi on lower-powered hardware (e.g. the Firefly-zero people) they seem to be very happy with Wasmi's performance so far.
@herobird thanks for the hard work and congrats with the release!!1
This is incredibly impressive. Great work!
Thank you! It means a lot coming from someone with your experience in the field.