In short. Prometheus is a worker that knows about all your other services. Each service of interest can expose an endpoint that prometheus scrapes periodically. So the services just say what the current state is, and prometheus, since it keeps asking, knows and stores what happens over time. Grafana is a web service that uses prometheus as a data source and can visualize it very nicely.
Prometheus also comes with an Alert Manager, where you can set up rules for when to trigger an alert, that can end up as an email or slack integration.
They are all very useful, and gives a much needed insight into how things are going.
This is fine, if you’re on AWS and can use AWS APIs. If you’re not (especially if you’re on bare metal), something like K8s can be nice.
> I found that to be quite impressive, for one person, one year, and would probably be completely impossible if it wasn't for k8s.
I've always found this interesting about web based development. I have no idea what Prometheus, grana, etc do. Ive never used k8s.
And yet, as a solo dev, I've written auto-scaling architecture using, for example, the AWS ec2 apis that let you launch configure and shutdown instances. I don't know what else you need.
Really the only advantage I see to morass of services is you get a common language so other devs can have a slightly easier time of picking up where someone left off. As long as they know all the latest bs.