OpenTelemetry at Scale: Using Kafka to handle bursty traffic

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by blinded on Sunday October 22, 2023 @08:13PM (#37980236)

This arch is how the big players do it at scale (ie. datadog, new relic - the second it passes their edge it lands in a kafka cluster). Also otel components lack rate limiting(1) meaning its super easy to overload your backend storage (s3).

Grafana has some posts how they softened the s3 blow with memcached(2,3).

1. https://github.com/open-telemetry/opentelemetry-collector-co... 2. https://grafana.com/docs/loki/latest/operations/caching/ 3. https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cl...

I know the post is about telemetry data and my comments on grafana are logs, but the arch bits still apply.

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by ankitnayan on Monday October 23, 2023 @01:19AM (#37981908)

Caching is to improve read performance whereas Kafka is used to handle ingest volume. I couldn't correlate the Grafana articles shared

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by blinded on Monday October 23, 2023 @11:42AM (#37987316)

Yep! Should have made that more clear. Brought it up as an example of other parts of the system that require scaling.

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by wardb on Monday October 23, 2023 @01:59AM (#37982093)

Grafana Labs employee here => On the linked articles: I'm not aware of any caching being used in the writing data to S3 part of the pipeline other then some time based/volume based buffering at the ingester microservices before writing the chunks of data to object storage.

The linked Loki caching docs/articles are for optimising the read access patterns of S3/object storage, not for writes.

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by blinded on Monday October 23, 2023 @02:49PM (#37989921)

Yes. Thanks for the reply!

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by kilotaras on Monday October 23, 2023 @05:04AM (#37983239)

Meta is also using something similar, see 4.1.1 of "Meta’s Next-generation Realtime Monitoring and Analytics Platform" [0]

https://www.vldb.org/pvldb/vol15/p3522-mo.pdf

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by blinded on Monday October 23, 2023 @11:42AM (#37987322)

Super cool!

[ Parent | on HN ]