News for hackers. Stuff that matters.
Re:Apache Flink by bunderbunder on Wednesday January 30, 2019 @10:32AM (#19035831)
This has been my experience, too. I worked at just one place that had a really good handle on high-volume, high-velocity streaming data, and they didn't use Flink or Storm or Kafka or anything like that. They mostly just used the KISS principle and a protobuf-style wire format.[1]

There is definitely a point where these sorts of scale-out-centric solutions are unavoidable. Short of that point, though, they're probably best avoided.

[1]: (It's truly amazing how many CPU cycles you can reclaim just by removing branch instructions from your message deserialization code.)

Re:Apache Flink by mikhailfranco on Friday February 01, 2019 @05:36AM (#19053112)
See McSherry et al, "Scalability! But at what COST?"

https://www.usenix.org/system/files/conference/hotos15/hotos...

Re:Apache Flink by lenticular on Wednesday January 30, 2019 @12:33PM (#19037202)
That's a smart way of doing it. In much ML work, I've found that the hash trick can be used to train many more things in a constant memory, out-of-core fashion than most people think.