This why java seems an odd choice; I can't imagine trying to run a codebase that's a split between assembly and java (keeping behaviour the same as more and more code is shifted).
The best idea I can see for that, is start with an interpreter for the assembler in java, moving procedure calls "up" into java.
But I imagine getting the interpreter good enough to run the same as the original target machine would be a huge task in itself.
[ed: leveraging clojure on the first iteration(s) might actually make the project feasible, though...]
I imagine you might get the interpreter running up to spec by flipping actual mainframe to debug, then writing a supervisor at the interpreter's side that checks for parity at each instruction, then single-steps the physical machine.
Yes, in that projects which try to reimplement large existing systems from the ground up by nailing down the requirements and working from there, almost invariably fail (even if they formally “succeed” in the sense of being accepted and then facing years of remediation.)
You always want to do a Ship of Theseus replacement rather than all-at-once, if possible, and finding a way to lift-and-shift the existing implementation to a platform that supports the way you'd like to replace thing in the long term is a way of getting as close as possible to that when you can't practically do it directly.