OpenTelemetry at Scale: Using Kafka to handle bursty traffic

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by bushbaba on Sunday October 22, 2023 @06:01PM (#37979488)

Seems like overkill no? Otel collectors are fairly cheap, why add expensive Kafka into the mix. If you need to buffer why not just dump to s3 or similar data store as a temporary storage array.

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by francoismassot on Sunday October 22, 2023 @06:11PM (#37979547)

I really like this idea. And there is an OTEL exporter to AWS S3, still in alpha but I'm gonna test it soon: https://github.com/open-telemetry/opentelemetry-collector-co...

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by prpl on Sunday October 22, 2023 @06:22PM (#37979617)

Why not both, dump to S3 and write pointers to kafka for portable event-based ingestion (since everybody does messages a bit differently)

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by bushbaba on Sunday October 22, 2023 @06:46PM (#37979765)

No need as s3 objects is your dead letter queue and the system should be designed anyway to coupe with multiple write of same event.

The point is to only use s3 etc in the event of system instability. Not as a primary data transfer means.

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by lmm on Sunday October 22, 2023 @07:22PM (#37979976)

> If you need to buffer why not just dump to s3 or similar data store as a temporary storage array.

At that point it's very easy to sleepwalk into implementing your own database on top of s3, which is very hard to get good semantics out of - e.g. it offers essentially no ordering guarantees, and forget atomicity. For telemetry you might well be ok with fuzzy data, but if you want exact traces every time then Kafka could make sense.

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by dikei on Sunday October 22, 2023 @10:25PM (#37980935)

Yeah, and to use S3 efficiently you also need to batch your messages into large blobs of at least 10s of MB, which further complicates the matter, especially if you don't want to lose those messages buffers.

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by bushbaba on Sunday October 22, 2023 @11:54PM (#37981443)

if your otel collector is being overwhelmed. In such cases you have a lot of backlogged data not able to be ingested. So you dead letter queue it to s3 for freeing up buffers.

The approach here is to only send data to s3 as a last ditch resort.

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by lmm on Tuesday October 24, 2023 @01:48AM (#37995206)

If you're ok with losing some data when your collectors are overwhelmed, surely you'd just drop overflowing data in that case? Why go to all the effort of building a fallback ingestion path if it's not going to be reliable?

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by ankitnayan on Monday October 23, 2023 @02:22AM (#37982201)

it's very hard to think s3 work as a buffer. Every datastore can work for almost all storage usecases buffer/queue/db when the scale is low but the latter were designed to work at scale

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by richieartoul on Monday October 23, 2023 @01:15AM (#37981885)

(WarpStream founder)

This is more or less exactly what WarpStream is: https://www.warpstream.com/blog/minimizing-s3-api-costs-with...

Kafka API, S3 costs and ease of use

[ Parent | on HN ]

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by lnenad on Monday October 23, 2023 @09:05AM (#37985055)

Love the website, looks good, clear and to the point.

[ Parent | on HN ]