Agreed on the “whoa moment”. Not needing health checks seems pretty compelling. I’m curious if there are any off-the-shelf pull-based load balancer that don’t require the full HTTP request copied onto the queue before handing off to a worker?

You need health checks. If not for LB then for your own monitoring.

you don't

a server can return 200 OK to every health check query and 5xx to every production request, is that server up or down? hopefully clear it is down

similarly, if it returns 5xx to every health check query but 200 to every prod request, is it up or down? hopefully clear it is up

if you want to monitor the status of a server (or application) for internal purposes, that's fine, but that's (at best) supplementary signal for the decisions made by a load balancer, not something you can make load balancing decisions on in isolation

You still need health checks, though? Otherwise, how do you tell the difference between: "no traffic + server alive" vs "some traffic + server is dead". Yeah, you can monitor throughput on a load balancer, but if I ever again wake up from an alert about not traffic being served - I will throw hands.

Yeah, they’d have to exist but it would be for quite a different purpose. I wonder if that would mean we could implement them in different ways, e.g have a health check service that everything pings and if a ping isn’t received for N minutes, assume it’s dead and trigger some replacement routine or alert.

This begins to look a lot more like a software watchdog at that point, and you can even have each service provide a count of outstanding/processed requests per tick and if a server never gets outstanding or processed requests you could have the watchdog kill it off.

Your server scaling system will need healthchecks. But:

1. It can be more tolerant of failures, since an unresponsive server doesn't mean failed requests.

2. Hooking up your server scaling system to HAProxy, etc. isn't the easiest thing in the world anyway.

server health is necessarily a function of the actual production traffic it receives, determined at the application layer, as observed by a specific observer

it can't be known by the server itself, as (among many other reasons) the server can't know about network issues between itself and any upstream caller

it can't be determined by out-of-band health check queries, because those queries don't represent actual traffic, the simplifying assumption that they _do_ introduces many common failure modes that any seasoned engineer can speak at length about

health checks can be a nice additional signal on top of monitoring actual prod traffic, but they can't be used by themselves, they just don't capture enough relevant information