News for hackers. Stuff that matters.

A comment on OpenTelemetry at Scale: Using Kafka to handle bursty traffic. Parent comment

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by bushbaba on Sunday October 22, 2023 @11:54PM (#37981443)
if your otel collector is being overwhelmed. In such cases you have a lot of backlogged data not able to be ingested. So you dead letter queue it to s3 for freeing up buffers.

The approach here is to only send data to s3 as a last ditch resort.

[ Parent | on HN ]
Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by lmm on Tuesday October 24, 2023 @01:48AM (#37995206)
If you're ok with losing some data when your collectors are overwhelmed, surely you'd just drop overflowing data in that case? Why go to all the effort of building a fallback ingestion path if it's not going to be reliable?
[ Parent | on HN ]