Authentication is a good basic example. Different APIs will have different requirements for authentication headers. Successful authentication frequently requires multiple steps, like an initial request to get a session token. Many APIs will also require your request to be signed according to their specifications, and to stuff that signature in a header with a special name.

Every API expects to receive data in its own format. I don't want to have to make a bunch of validators that lay on top of your data model. These can and should be provided, and Swagger makes that automatic.

As eropple states, if something is making me do all of that by hand, I'm already looking for something else that will allow me to just say "import project_lib; project_lib.authenticate(); ..."

If you're just talking about something that is read-only and that only emits a single field I care about, sure, no need for a library. Something like the free version of ipinfo.io would be a good example.

Stuff that's more complicated than that, yes, it needs a client library.

Any time you have a statically-typed language consuming you. Having to write my own Jackson declarations to pull in your API to a JVM project or my own DataContract barf for a CLR one is a quick way to make me hate you, and me hating you means I'm already looking for an alternative that isn't you that gets out of my way.