Re:Apache Flink
by mikhailfranco on Friday February 01, 2019 @05:36AM
(#19053112)
See McSherry et al, "Scalability! But at what COST?"
https://www.usenix.org/system/files/conference/hotos15/hotos...
Re:Apache Flink
by lenticular on Wednesday January 30, 2019 @12:33PM
(#19037202)
That's a smart way of doing it. In much ML work, I've found that the hash trick can be used to train many more things in a constant memory, out-of-core fashion than most people think.
There is definitely a point where these sorts of scale-out-centric solutions are unavoidable. Short of that point, though, they're probably best avoided.
[1]: (It's truly amazing how many CPU cycles you can reclaim just by removing branch instructions from your message deserialization code.)