Kafka at the Low End

Re:Kafka at the Low End by stickfigure on Tuesday February 18, 2025 @11:49PM (#43098703)

If that's true, you managed to do much better than these folks:

Maybe you should write a letter?

[ Parent | on HN ]

Re:Kafka at the Low End by winrid on Wednesday February 19, 2025 @12:44AM (#43098970)

It's possible the person you're replying to wasn't using replication, so it's entirely different. Those folks also used "synchronous_commit is set to remote_write" which will have a performance impact

[ Parent | on HN ]

Re:Kafka at the Low End by pritambaral on Friday February 21, 2025 @08:30AM (#43127177)

This is correct. My use-case was safe with eventual consistency, so I could've even used `synchronous_commit=off`, but I kept it to 'local' to get a baseline. Was happy with the 60k number I got, so there was no need for 'off'.

But I think the biggest reason I hit that number so easily was the ridiculous ease of batching. Starting with a query to select one task at a time, "converting" it select multiple tasks instead is ... a change of a single integer literal. FOR UPDATE SKIP LOCKED works the same regardless of whether your LIMIT is 1 or 1000.

[ Parent | on HN ]