Re:Ruins of forgotten empires: APL Languages (2013)
by nickpsecurity on Saturday June 13, 2015 @10:58AM
(#9711190)
I agree it's important to consider refrigeration costs. Yet, the rest seems hypothetical given their site says it scales to petabytes across many machines, from clusters to grid computing. Their benchmarks are on a diverse array of machines with some being flash storage arrays and others compute nodes with 16+ cores & 512GB memory. I have a hard time believing this thing is going to choke on most datasets.
Licensing and hardware will probably be the biggest costs here.
Re:Ruins of forgotten empires: APL Languages (2013)
by rbanffy on Saturday June 13, 2015 @08:47PM
(#9713370)
> I have a hard time believing this thing is going to choke on most datasets.
It won't, but 512GB is nowhere near "big data". In fact, most datasets we see should not be called that. My personal rule of thumb is that nothing that can fit in less than a dozen machines can be called "big data". The whole idea begun when we started manipulating datasets that were so huge that moving the data from where it was stored to where it would be processed was no longer practical.
Re:Ruins of forgotten empires: APL Languages (2013)
by nickpsecurity on Sunday June 14, 2015 @01:28AM
(#9713810)
My comment said "petabytes" of data across many machines. The 512GB RAM was one node they tested it on. As in, the 512GB is around how much can be in one node at once rather than what the database can process. If petabytes of OLAP don't count, I'll concede that this database isn't for "big data."
64-bit processors and more than 4 gigs of RAM are relatively new things. Mmap is useless for datasets larger than the disk you can attach to the CPU mmapping it or the size of its virtual address space. In 1991 it would be OK to call a couple billion entries "big data" (although we were yet to invent the term) but not anymore.
Having said that, the heat dissipation of instantiating objects from their on-disk representation is probably a relevant part of the refrigeration costs of any data center. We just have unimaginable computing power available and, sometimes, we do things the easy way, not the most efficient one.