The API here is HTML + HTTP. That's it, that's the API. In a browser, the Hacker News website/HTML is not the API.

"I want to get the data and compute some statistics on it. For that I need to know the structure of the data in advance, need to know which URLs to retrieve, which operations to perform, which data structures to expect"

That's not something that was "promised" by REST. Quite the opposite, REST is about applications not needing to know the structure that's on the other side, only the format and how to navigate it.

"So what am I still not getting?"

My assumption, but since you asked: I think you're expecting the original REST paper and hypermedia to be something that it isn't, and to apply to things it isn't supposed to. You're trying to apply it to arbitrary APIs that are supposed to be consumed exclusively by software but don't really match its model or intent.

Perhaps you're expecting hypermedia to replace something else? You're mentioning "WSDL, OpenAPI and friends". Well, those are perfectly fine, but they're orthogonal to REST and hypermedia.

The H of HATEOAS means hypermedia, a term created by Ted Nelson in the 60s. It's related to hypertext, which is the HT of HTML and HTTP. HATEOAS/REST is about creating hypermedia APIs, including HTML/HTTP. This means rich context, like Wikipedia, a database, or a Website.

Not everything matches this. The app I work on doesn't. And that's completely fine.

The goal of the paper is to define clients that are 100% decoupled from the applications being served by some server. The browser is of course the best example of that. The only "shared knowledge" is in HTTP and in the format (HTML, OData, AtomPub). Excel consuming arbitrary OData is another example, Excel only knows HTTP and OData, and it's able to consume arbitrary data sources.

If you want to have, say, a more user-friendly iPhone banking app, that's probably not gonna be an hypermedia app, it's just a regular application that's specific to your bank. This is not decoupled! Which is 100% fine, especially for user friendliness. For this you can start hardcoding URLs and baking in some assumptions, customizing some screens in the app. All of that is fine, but it's beyond the scope of "pure" hypermedia. Which is fine.

As for:

> My assumption, but since you asked: I think you're expecting the original REST paper and hypermedia to be something that it isn't, and to apply to things it isn't supposed to. You're trying to apply it to arbitrary APIs that are supposed to be consumed exclusively by custom software.

I just want to point out that Fielding in his thesis details many different types of architectures. His focus is on REST and hypermedia - and especially how it fits well with efficient and correct caching.

But he talks about movable code (java applets, or essentially also modern JavaScript applications like those built around React), RPC etc.

I think the thesis remains relevant exactly because it's not only about REST.

I think you have some degree of confusion on what REST is in general, and HATEOAs in particular.

"Which URLs to retrieve" is provided by the hypermedia format you requested for a particular resource. There are many HATEOAS-based specifications for formats. Your client asks for the ones it supports when sending a request,and the server delivers one based on your request and what it supports.

"Which operations to perform" is also specified as part of the hypermedia format you requested. Your format specifies link relationships valid for a resource according to it's current state.

"Which data structure to expect" makes no sense and is a reflection of the degree of confusion you're in. Your client is responsible for specifying which media type it can consume. The "data structure to expect" is specified as part of the media type. If your client expects specific data structures, it needs to support specific media types that represents them. These media types are not application/json or application/xml. They are media types designed to reflect the format used to express your resource.

> what can I do except baking the structure into my code in one way or another?

You're missing the whole point. With REST, the "structure" that you "bake into your code" is not the URLs, operations, or data formats. You only need a single root resource as the entry point, and that resource expresses all link relations that you need to navigate towards your resource. The link relations provided by the resource already specify which other resources are available to your client, which also includes operations they currently support. Your client also has the responsibility of requesting the resources it wants expressed in the format it supports, which included the data format. I mean, if you write a Atom client you don't request application/xml. Instead, you request application/atom+xml.

If you need an analogy to understand the whole point, check how you navigate HackerNews with a browser. You open a resource representing the main page, and you're presented with a list of resources which include a collection of submissions, links to other collections reflecting specific queries such as new discussions, links to resources that allow you to login/logout or represent your user account. From that initial resource, you use the links provided in it to request other resources. If your current application state supports logging in, there will be a link to that. If your state is already logged in, there will be resources made available only in that state such as your account or logging out. If you opt to request a resource representing a submission, you'll get the submission text and/or link, along with comments. Each comment provides links to other resources such as individual comments, etc.

That's basically the whole point of REST. Do you hardcode references to each discussion or comment? No. you get them through links provided by other resources. Do you always get access to the resource representing your user account? No, that depends on the application state. Etc etc etc.