News for hackers. Stuff that matters.
Re:Grafana Tempo, a scalable distributed tracing system by RhodesianHunter on Wednesday October 28, 2020 @03:07PM (#24922850)
Even with batching from Tempo, wouldn't that cost many thousands per month in S3 PUT costs alone?
Re:Grafana Tempo, a scalable distributed tracing system by number101010 on Wednesday October 28, 2020 @03:44PM (#24923292)
We batch up traces in a block and write a block at a time. Internally we are currently configured to write 100k traces in one batch.
Re:Grafana Tempo, a scalable distributed tracing system by atombender on Wednesday October 28, 2020 @08:34PM (#24925806)
Doesn't this cause explosive memory usage? What happens if there's some congestion? Is there a circuit breaker to start dumping (discarding) log entries past a certain limit?

I was testing Google Pub/Sub's Go client for publishing internal API event data for later ingest to BigQuery, and it turns out Pub/Sub publishing is not that much faster than writing directly to BigQuery. The buffer sizes we'd need to avoid adding latency to our APIs would have to be ridiculously high; the Pub/Sub client buffers and submits batches in the background (its default buffer size is 100MB!). I don't like the idea of having huge buffers that increase with the request rate.

Conversely, pushing the data to NATS in recent time without any buffering or batching turned out to be fast enough to not add any latency. You have to be able to receive messages very fast on the consumer side (as NATS will start dropping messages if consumers can't keep up), but you can simply run a few big horizontally autoscaled ingest processes that can sit there ingesting as fast as they can, which never impacts API latency at all.

Re:Grafana Tempo, a scalable distributed tracing system by marcinzm on Wednesday October 28, 2020 @06:25PM (#24924827)
S3 PUTs are $0.005 per 1000. If you're writing twice a second that comes out to $25/month.
Re:Grafana Tempo, a scalable distributed tracing system by RhodesianHunter on Thursday November 05, 2020 @12:12PM (#24999903)
Yeah, but the person I'm responding to is suggesting 170k+ spans per second so how is twice per second relevant?