Re:Three years later, Mr. Moore is still letting us punt on ...
by adgar on Friday January 27, 2012 @05:17PM
(#3520441)
> Also, don't underestimate how large log files can grow in a data-driven business (like AirBNB seems to be). I could easily believe that they have many terabytes of data just from logging actions their customers have taken.
Logs don't have remotely the same access requirements as the databases used to serve a product.
Re:Three years later, Mr. Moore is still letting us punt on ...
by j_baker on Friday January 27, 2012 @05:26PM
(#3520484)
Indeed, but it's worth pointing out that in this case "different" doesn't necessarily imply "easier". Instead of having to access the data across many concurrent connections, you have to be able to store the data efficiently so that it doesn't take up too much space and you can do jobs on them that don't take 3 weeks to complete. And let's not get into how you collect and merge them together. There are open source tools to do these things, but you're still looking at a decent amount of infrastructure to make it work.
Also, don't underestimate how large log files can grow in a data-driven business (like AirBNB seems to be). I could easily believe that they have many terabytes of data just from logging actions their customers have taken.