It's flexible for some architectures, but not others. If your potential customer runs their own servers or something like k8s that they can deploy Prometheus to, sure. But what if your potential customer runs on a PaaS like Heroku? Or a FaaS like lambda? Or what if what they primarily care about getting metrics from are jobs on some managed data processing service, like cloud dataflow? Etc.
In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you.
maybe it would be beneficial to replace it or strip it down to ensure your customer never has
to worry about storage performance, for example.
The way the user runs it means that they don't store any metrics locally. The Prometheus agent on the cluster does scraping and then sends the data immediately up to the central SaaS instance (Cortex). what if your potential customer runs on a PaaS like Heroku
The range of exporters is very extensive (https://prometheus.io/docs/instrumenting/exporters/). And, as K8s has become so important lots of other people are publishing Prometheus metrics. In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players
like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code
against to push data to you.
Cortex provides a scalable backend storage for Prometheus metrics, it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples.In Weaveworks case we've build integrations into our product that help connect and pull data from other systems. Overall I think Prometheus metrics have a strong future, and there will be lots of SaaS options for sending and consuming the data.
Ah, that's great! I was mistakenly assuming that Prometheus would still have to write the data to disk; I didn't realize you could disable local storage.
> The range of exporters is very extensive
Sure, but that doesn't solve the problem that your heroku dyno, or lambda function, etc cannot expose metrics for Prometheus to pull, but it could easily push metrics somewhere.
> it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples.
That's fair; I was thinking of Cortex as something to adopt off the shelf, instead of something to integrate with. However, as a SaaS provider you can always build more custom functionality alongside Cortex instead of making it a direct part of the open-source project.
You are correct: there is no option to disable so we set to discard after 2 hours. Prometheus doesn’t write continuously to disk so the io is not noticeable.
To your original question, yes we could produce another agent from the Prometheus code which did discovery, scraping and sending but not disk storage. We choose not to as it’s another thing to maintain.
Also we want an option to queue on disk when the network link to push is down.
(I am a Cortex maintainer)
Thats a good point; Cortex does accept pushes of metrics but its not that simple, and its definitely not easy right now. We're going to be working with the OpenMetrics project to have a better story about push and serverless - I expect we'll have this pretty soon, ie a matter of months.
If you want a sneak peak, checkout grafana/loki; its based on Cortex and has all theses features - buts it for logs, not metrics.
It is my understanding that Prometheus's storage engine is LSM based, and uses a compression scheme inspired by gorilla.
Boltdb is basically a btree which causes more write amplification.
Maybe you should try badgerdb.
After compressing timeseries they are stored and indexed using a database-like thing, typically DynamoDB or Bigtable, but soon BoltDB will be an option.
(I am a Cortex maintainer)
I played with Cortex for a bit, built my own gateway to match & assign user IDs for incoming Prometheus push requests but from operational perspective running Cortex can be very expensive (if you want it just as a side project) :) GCP Bigtable seems like the best way to store & query data but even the dev instance is prohibitively expensive (last time I checked it was ~1k USD)