News for hackers. Stuff that matters.
Re:Optimizing Conduit – 5x the Throughput by raulb_ on Wednesday March 12, 2025 @07:25AM (#43342075)
Meroxa improved the Conduit streaming engine by redesigning its architecture, which increased throughput by five times while simplifying the code. The new single-threaded approach processes batches of records, reducing latency and enhancing performance significantly compared to the old system. Testing showed that the new architecture can handle a peak of over 500,000 messages per second, demonstrating its efficiency with various message sizes.