Re:Polar Signals Cloud Is Generally Available
by brancz on Wednesday October 11, 2023 @01:29AM
(#37841257)
Yeah that’s basically how this system is built as well. We buffer a certain amount of data in the ingestion nodes and then offload it to object storage. In object storage it’s actually laid out in the parquet format so we could allow users to export/download this. Or like I said, the object storage cost is really the smallest piece in the system we might even just decide not to have any retention at all.
I’ve enjoyed having 3-5 years of prom metrics retention to look at seasonal traffic trends but we spent a fair bit of CPU aggregating raw metrics down to the right granularity for that kind of long term retention. My feeling is that the observability world is moving towards small, localized installations with real-time data and separate systems with cheaper, slower, but long term retention. Curious how you see it as you’re building a product in the space.