News for hackers. Stuff that matters.
Re:OpenTelemetry in 2023 by singron on Monday August 28, 2023 @12:00PM (#37296191)
I recommend sampling traces if you aren't. I've been unimpressed with datadog apm, which has no affordable configuration. We've been running our own Jaeger stack with 0.1% sampling, and it's negligible to run compared to datadog apm.

For metrics and logs, sampling isn't so useful, so I don't have a good answer. Datadog has 80% gross margin, so at most 20% of what you pay them is the infra, so you stand to save a lot of money running your own open source stacks if your labor costs would be less than that 80%. With datadog, we have a project every 3 months to reduce usage, so it's not like we aren't constantly babysitting it anyway.

Re:OpenTelemetry in 2023 by phamilton on Monday August 28, 2023 @01:35PM (#37297670)
> I've been unimpressed with datadog apm, which has no affordable configuration

FWIW we sample in datadog APM and it works fine to control costs, I'm not sure what issues you hit.

Re:OpenTelemetry in 2023 by pranay01 on Monday August 28, 2023 @03:09PM (#37299007)
If you’re looking for open source APM stack which is OpenTelemetry native and you can self host - you can check out SigNoz ( https://github.com/SigNoz/signoz)
Re:OpenTelemetry in 2023 by paulddraper on Monday August 28, 2023 @04:37PM (#37300100)
> datadog apm, which has no affordable configuration

In my experience, Datadog's ingestion sampling works pretty well.

And there's retention filters you can use to override.