Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic
by dikei on Sunday October 22, 2023 @10:25PM
(#37980935)
Yeah, and to use S3 efficiently you also need to batch your messages into large blobs of at least 10s of MB, which further complicates the matter, especially if you don't want to lose those messages buffers.
[ Parent
| on HN ]
Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic
by bushbaba on Sunday October 22, 2023 @11:54PM
(#37981443)
if your otel collector is being overwhelmed. In such cases you have a lot of backlogged data not able to be ingested. So you dead letter queue it to s3 for freeing up buffers.
The approach here is to only send data to s3 as a last ditch resort.
Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic
by lmm on Tuesday October 24, 2023 @01:48AM
(#37995206)
If you're ok with losing some data when your collectors are overwhelmed, surely you'd just drop overflowing data in that case? Why go to all the effort of building a fallback ingestion path if it's not going to be reliable?