This is the same approach taken by Lua and many of the original Pascal compilers, which had to run on very slow hardware, so there's pretty good precedent that you can get adequate performance.
> But you're also using bytecode, which involves more caring about speed, so you've got a bit of a mismatch
I wouldn't describe it as a mismatch as much as it is a trade-off. Because the compiler only has a peephole view into the source when it needs to generate code, many optimizations are off the table. However, because we have full control over the instruction set, we can sometimes tweak the bytecode format in order to more naturally align with the compiler.
In return for not needing an intermediate representation or AST, we get a much simpler compiler, especially in C where you have to worry about memory management.
Optimization along any axis (code size, simplicity, compile-time performance, runtime performance, memory size, etc.) always involves some level of violating software engineering norms.
One way to think of software engineering is that it's the practice of optimizing long-term developer velocity. Any practice that optimizes other factors generally does so at the expense of that. That's OK. Meta-software engineering is about knowing when long-term maintainability is the right factor to optimize for. :)
Technically, it's perfectly fine for a VM to be designed so that the semantic gap is small between it and the intended language family.
But you're also using bytecode, which involves more caring about speed, so you've got a bit of a mismatch - as your code tries to match the multiple layers you're skipping, your code will get more complicated or your VM gets more arbitrary, so of the situation you have here.