News for hackers. Stuff that matters.
Re:Ruins of forgotten empires: APL Languages (2013) by nickpsecurity on Friday June 12, 2015 @07:18PM (#9709144)
Another person realizing my meme that IT regularly forgets and (when lucky) re-invents the past. Often in an inferior way. K and J were great tools for datamining. Well-implemented, too. Hadoop-style efforts could've saved a lot of time and gained plenty efficiency starting out with such a design. Any suggestions on best way to solve this? A Wikibooks-like, categorized, searchable site of all kinds of solutions to various problems, perhaps?

Fortunately we have nice counter-examples with groups such as FoundationDB doing Hadoop scale with strong consistency. Building on the right, proven principles can get one ahead.

Re:Ruins of forgotten empires: APL Languages (2013) by rbanffy on Saturday June 13, 2015 @07:11AM (#9710662)
What I find slightly disturbing is that people remember history but forget we got to where we are precisely because of it.

64-bit processors and more than 4 gigs of RAM are relatively new things. Mmap is useless for datasets larger than the disk you can attach to the CPU mmapping it or the size of its virtual address space. In 1991 it would be OK to call a couple billion entries "big data" (although we were yet to invent the term) but not anymore.

Having said that, the heat dissipation of instantiating objects from their on-disk representation is probably a relevant part of the refrigeration costs of any data center. We just have unimaginable computing power available and, sometimes, we do things the easy way, not the most efficient one.

Re:Ruins of forgotten empires: APL Languages (2013) by nickpsecurity on Saturday June 13, 2015 @10:58AM (#9711190)
I agree it's important to consider refrigeration costs. Yet, the rest seems hypothetical given their site says it scales to petabytes across many machines, from clusters to grid computing. Their benchmarks are on a diverse array of machines with some being flash storage arrays and others compute nodes with 16+ cores & 512GB memory. I have a hard time believing this thing is going to choke on most datasets.

Licensing and hardware will probably be the biggest costs here.

Re:Ruins of forgotten empires: APL Languages (2013) by rbanffy on Saturday June 13, 2015 @08:47PM (#9713370)
> I have a hard time believing this thing is going to choke on most datasets.

It won't, but 512GB is nowhere near "big data". In fact, most datasets we see should not be called that. My personal rule of thumb is that nothing that can fit in less than a dozen machines can be called "big data". The whole idea begun when we started manipulating datasets that were so huge that moving the data from where it was stored to where it would be processed was no longer practical.

Re:Ruins of forgotten empires: APL Languages (2013) by nickpsecurity on Sunday June 14, 2015 @01:28AM (#9713810)
My comment said "petabytes" of data across many machines. The 512GB RAM was one node they tested it on. As in, the 512GB is around how much can be in one node at once rather than what the database can process. If petabytes of OLAP don't count, I'll concede that this database isn't for "big data."
Re:Ruins of forgotten empires: APL Languages (2013) by MrBuddyCasino on Friday June 12, 2015 @07:53PM (#9709258)
Yeah, a resource of canonical answers to questions. Often arguments are repeated again and again in discussions, often poorly. If you could just point people at this, and pros and cons would become better over time, I think that would be a real win.
Re:Ruins of forgotten empires: APL Languages (2013) by Blackthorn on Friday June 12, 2015 @11:54PM (#9709807)
I don't know. Seems like the sort of thing that would be useful for a good few years and get popular, but then new questions and resources would start getting closed and locked for being off-topic or irrelevant...Wink, wink.
Re:Ruins of forgotten empires: APL Languages (2013) by nickpsecurity on Saturday June 13, 2015 @10:52AM (#9711166)
That's exactly what I'd try to avoid. You could say all topics are shallow there and we'd be the deep thinkers. ;)