and most of the time you can't handle it anyway. I mean consider you are having a database query and it fails because the connection error'd (network split). what to do know? restart the network switches and wait? of course not in http you will just print a 5xx err and hope it comes back. in go you need to bubble up these errors to your middleware and handle it there.

If this is seriously what you do when something underneath you fails, then you aren't writing in the domain of software where errors really matter, and you're probably better off in a language that has a gigantic try/catch handler around everything. In that case you may as well just log (or not) and ignore the error in Go as well then.

At high scale, you want to perhaps try your query a few more times, backing off exponentially, maybe even round-robining (or more complicated load balancing) among different targets. Throwing up your hands and 5xx-ing a request that easily is not something you can afford at high scales of traffic where small outages happen _all the time_.

> At high scale

so basically you mean that this only applies to like 0.001% of all websites/apps.

thus it's stupid to do it. I mean even than you would have an error that you need to bubble up or circuit break it somwhere but probably not at this layer.

also KISS and MVP, never ever overcomplicate something when you do not know anything yet, so most people don't need to built for scale at day 1 and it also is stupid to do it.

You can get off-the-shelf libraries that do this for you with zero-effort sane defaults. Most of them let you tweak the settings. I'm talking about Python's `requests` and Rust's `reqwest` here btw, two of the most popular HTTP client libraries out there. Python's `urlopen3`, which `requests` uses, exposes options for backoff strategies. `aiohttp` has a lot of these parameters as well. Unfortunately Rust's equivalent, `hyper`, isn't nearly as ergonomic/easy to use.

That’s something new handle as close as possible to the outbound client, which is several layers in from the inbound request. Much of it is in gRPC middlewares, so by the time a gRPC client returns an error to the gateway it’s already over.

At the very least, you have to log that error.

Our coding convention (and I happen to agree) is that the outermost layer to touch the error is the one to log it. In this case the HTTP handler, or perhaps a gRPC middleware. You don't want to be logging errors as you propagate them, or the logs will show the same error at a bunch of different call sites vs. one complete picture of what happened.