News for hackers. Stuff that matters.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by infinite8s on Tuesday April 27, 2021 @03:58PM (#26960726)
OK thanks for the insight. I havenv't been tracking arrow-flight recently, but it does seem more geared to an RPC use case than one way streaming.

I've been playing with pushing arrow batches using (db -> server-side turbodbc -> network -> browser -> webworker using transferable support), leveraging a custom msgpack protocol over websockets, since most databases don't support connectivity from a browser. Not dealing with big data by any means (max a million records or so), and so far it's been fairly performant.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by lmeyerov on Tuesday April 27, 2021 @04:16PM (#26960949)
yep that's pretty close (and cool on turbodbc, we normally don't get control of that part!)

for saturating network, afaict, it gets more into tuning around browser support for parallel streams, native compression, etc. we've actually been backing off browser opts b/c of unreliability across users, and thus focus more on what we can do on the server..