The app with the twisted logic (where bugs take weeks to fix, that has mutated over years and years of hacked-in change requests and quirky tweaks often for specific users' needs) is the one that screams for rewriting, but this is also the app where no-one knows exactly what it does anymore.
And if you don't know what it does, I have no idea how you could claim to know what it should do.
Many of the odd features are there for customers who aren't even customers anymore, but no one knows. Even the code doesn't always help much -- I've seen processing where it fails in the mid-method every time for some foolish reason (and logs an error then returns), but the code after the failure is actually direly broken and must never be executed.
I've tried (failed, actually) rewriting one of these as well as doing lots of refactoring on these kinds of projects.
With the first kind of app, you still understand it, so bugs can be fixed, new features are possible, and there's no clear case to rewrite it. With the second kind of app, you can either do what I mentioned above (and maybe you're talking about this, more than a rewrite?) -- start talking with/watching users, take a lot of notes, and build a new similar application all the way through, which will necessarily start off missing much of the functionality of the existing one.
Refactoring isn't about giving you time; it's about paying off technical debt and forcing the codebase back into a state where you can do progressively larger refactorings. This includes modularizing the existing app so that you can replace it piecemeal. New pieces do not need to operate within the old framework. It's quite possible (and sometimes desirable) to refactor the original codebase completely out of existence.
First, a huge Java app is about 3 to 10 times smaller (in LoC) when ported to Ruby or Python.
Second, there is a lot of boilerplate code and configuration in a typical Java app. Not nearly that much with either Rails or Django.
And third, basic CRUD functionality, something that has tons of Java boilerplate associated with it, is usually very simple with Rails (or Django).
As for the tangled logic, it's the way many apps evolve. They are designed at their start and then refactored and patched into their current form. The current app wasn't designed anymore - it evolved - and, in order to properly maintain it, you'll have to understand how and why its evolution took place.
One more point - in order to reimplement it on top of a more modern framework, you don't need to understand thoroughly what your mutated-beyond-recognition Java app does - you only need to know what it should do. The fatal flaw you point out may not exist.
If you don't know what the app should be doing, then I have to agree - you are doomed from the start.
Refactoring will give you some time, as it reverses the mutations back into a designed state - and improve things a bit, but it will not be able to bring the orders-of-magnitude improvements you see when migrating from an old and obsolete (but still widely used) infrastructure, to a more modern one. That's when a rewrite would be warranted.