The goal of the paper is to define clients that are 100% decoupled from the applications being served by some server. The browser is of course the best example of that. The only "shared knowledge" is in HTTP and in the format (HTML, OData, AtomPub). Excel consuming arbitrary OData is another example, Excel only knows HTTP and OData, and it's able to consume arbitrary data sources.
If you want to have, say, a more user-friendly iPhone banking app, that's probably not gonna be an hypermedia app, it's just a regular application that's specific to your bank. This is not decoupled! Which is 100% fine, especially for user friendliness. For this you can start hardcoding URLs and baking in some assumptions, customizing some screens in the app. All of that is fine, but it's beyond the scope of "pure" hypermedia. Which is fine.
As for:
> My assumption, but since you asked: I think you're expecting the original REST paper and hypermedia to be something that it isn't, and to apply to things it isn't supposed to. You're trying to apply it to arbitrary APIs that are supposed to be consumed exclusively by custom software.
I just want to point out that Fielding in his thesis details many different types of architectures. His focus is on REST and hypermedia - and especially how it fits well with efficient and correct caching.
But he talks about movable code (java applets, or essentially also modern JavaScript applications like those built around React), RPC etc.
I think the thesis remains relevant exactly because it's not only about REST.
The API here is HTML + HTTP. That's it, that's the API. In a browser, the Hacker News website/HTML is not the API.
"I want to get the data and compute some statistics on it. For that I need to know the structure of the data in advance, need to know which URLs to retrieve, which operations to perform, which data structures to expect"
That's not something that was "promised" by REST. Quite the opposite, REST is about applications not needing to know the structure that's on the other side, only the format and how to navigate it.
"So what am I still not getting?"
My assumption, but since you asked: I think you're expecting the original REST paper and hypermedia to be something that it isn't, and to apply to things it isn't supposed to. You're trying to apply it to arbitrary APIs that are supposed to be consumed exclusively by software but don't really match its model or intent.
Perhaps you're expecting hypermedia to replace something else? You're mentioning "WSDL, OpenAPI and friends". Well, those are perfectly fine, but they're orthogonal to REST and hypermedia.
The H of HATEOAS means hypermedia, a term created by Ted Nelson in the 60s. It's related to hypertext, which is the HT of HTML and HTTP. HATEOAS/REST is about creating hypermedia APIs, including HTML/HTTP. This means rich context, like Wikipedia, a database, or a Website.
Not everything matches this. The app I work on doesn't. And that's completely fine.