Re:The Log: An Epic Software Engineering Article
by erikb on Saturday September 20, 2014 @06:44PM
(#8345792)
I'm a little sad that neither article (I've read some more beside the ones linked there) about mentions that the idea of logging here and processing there is pretty much the Unix principle of each program handling text (mostly lines of colon/comma separated data records) as input and creating another stream of text as output. Also I don't know if the idea of a central data storage (which is Krep's suggestion as far as I can understand) is really the solution to all our problems in a time of distributed everything.
[ Parent
| on HN ]
Re:The Log: An Epic Software Engineering Article
by fintler on Sunday September 21, 2014 @12:33AM
(#8346274)
Having several datacenters running Kafka with MirrorMakers between then (https://cwiki.apache.org/confluence/pages/viewpage.action?pa...) is far from being central data storage, it's more like a graph of (reasonably) ordered data distributed across the globe, which is available to any application.
I don't really see much of a relationship to unix text stream processing. Maybe in something like a Kafka -> Samza -> Kafka style system -- but even then, it feels like a stretch. There's much more of a focus on the ordering of messages rather than the concept of log here, process there.