Re:Building a Streaming Platform in Go for Postgres
by saisrirampur on Tuesday November 07, 2023 @02:13AM
(#38174059)
Thanks for the comment. We will aim to write a more deeper blog on this in the future.
We were observing 60<>40 ratio for Pull/Push, as specified in the blog and majority of the Push was target data-store (network) bound. So there was less room of optimization on the Go side.
The test was done with ~10-15K TPS on Postgres and the target was Azure Event Hubs, which has a limit of 1MB write batch size.
Re:Building a Streaming Platform in Go for Postgres
by earthboundkid on Tuesday November 07, 2023 @09:01AM
(#38176781)
Yes, thank you.
All valuable details were omitted, e.g. 1) how much data is being moved? a few megabytes? what is the state of art latency? maybe 0.001 second for that? when latency is still seconds huge, what can be further improved, in the go runtime or in the apps? etc
A highly efficient pull/push model without using go channel can easily beat your 1-5 seconds latency. A crappy implementation using channel can get worse.
Delay of several seconds is when you pass on information mouth to mouth, in 2023, that kind of delay doesn't justify a blog article on such fancy implementation. I'd suggest you to delete the implementation to rebuild from ground zero.