News for hackers. Stuff that matters.
Re:Google’s Dremel Makes Big Data Look Small by kamaal on Friday August 17, 2012 @04:48AM (#4395563)
A small note: Its great to see so many great tools coming up to solve the kind of problems which were earlier difficult/impossible to solve.

But however please check your bid data use cases many times before using big data tools. Because frankly 'big data' is becoming a just cool must use tool regardless of use cases people have these days. I've even seen data sizes as small as 10 MB being considered for bid data use cases. Often this gets subjected to a monstrously complex architecture for no good reason.

Generally most of these cases can be addressed and solved with as simple a tool like sqlite! And all you generally need is something like Perl with sqlite and ability to write simple SQL queries.

People get deceived very easily, When they look at GB scale XML files they think that is what big data is. Yet most of that generally and easily goes into a traditional RDBMS. And the performance is generally is in pretty acceptable limits. Mark up eats a lot of space and data size. When converted to flat file structures like csv's, tsv's and then imported to a RDBMS the data sizes are way smaller. I've some times seen an order of 10x difference.

Another annoying thing is abuse of NoSQL databases. Perfectly relational data is being de normalized, force fed in NoSQL databases and access data interfaces are generally bad buggy sub implementations of SQL.

This is almost like, people who don't understand SQL are condemned to implement it badly.

Re:Google’s Dremel Makes Big Data Look Small by wmf on Friday August 17, 2012 @12:08PM (#4397133)
See Nobody ever got fired for using Hadoop on a cluster: “A single ‘big memory’ (192 GB) server we are using has the performance capability of approximately 14 standard (12 GB) servers. … for about an eighth of the total cost.” http://research.microsoft.com/pubs/163083/hotcbp12%20final.p...

And The Seven Deadly Sins of Cloud Computing Research: "When designing a parallel implementation, its performance should always be compared to an optimized serial implementation, even if only for a small input data set, in order to understand the overheads involved. It is also worth considering whether distribution over multiple machines is required." https://www.usenix.org/conference/hotcloud12/seven-deadly-si...

Re:Google’s Dremel Makes Big Data Look Small by nrao123 on Friday August 17, 2012 @11:41AM (#4396942)
This is one of the most sensible comments I have ever read on Big Data. I sent it to a bunch of clients and people in large organizations and they vigorously nodding their heads.
Re:Google’s Dremel Makes Big Data Look Small by fkdjs on Friday August 17, 2012 @05:12AM (#4395609)
"Woops you want to query that column? Better wait 12 hours while we add an index to petabytes of data! Oh shoot, and that column too? Another 12 hours, and you have to use both columns in the where to use the index. "

If you are storing xml files for your column-oriented DB then you should be shot. I imagine protocol buffers or something like that might be smarter. IMO.

Your other stuff is way off base as well, but I'm lazy.

Re:Google’s Dremel Makes Big Data Look Small by garysieling on Friday August 17, 2012 @07:06AM (#4395797)
He's talking about people mis-applying big data tools for things which are clearly not big data - there are a lot of applications that don't have, or will ever have petabytes of data.

The point about XML is comparing the storage size on disk of the same data in an XML files vs the equivalent data in a database, not stored as XML in a database.

Re:Google’s Dremel Makes Big Data Look Small by fkdjs on Friday August 17, 2012 @01:06PM (#4397476)
Which is a false dichotomy, if you use the right storage mechanism for your column-oriented DB, the comparable relational DB storage will actually be larger since it has to index every column to provide the equivalent of what you can do with column-oriented storage. Also, another myth he brings up is that you can't have complex joins with column-oriented DBs. It's completely possible.
Re:Google’s Dremel Makes Big Data Look Small by garysieling on Friday August 17, 2012 @02:08PM (#4397825)
For practical purposes, you usually don't index every column. The index is a value-add for performance gains, which as far as I've seen requires custom implementation in map-reduce databases. When it is implemented, you would have the same storage problem. As an example you can look at Common Crawl, which has a public index of web page data. They provide a hadoop database of page source, and a smaller data set of page text. The page text database serves a similar function to an index. Using the text dataset instead of full HTML would be like an "Index Scan" in database optimizer terms.

I don't think he said you can't do complex joins; he said people tend to denormalize the data before putting it into NoSQL databases.

Re:Google’s Dremel Makes Big Data Look Small by fkdjs on Friday August 17, 2012 @11:23PM (#4400102)
If you compare the two, then you must compare apples with apples. That is, you must compare relational DBs where every column is indexed since with column-oriented DBs, you can search by arbitrary columns without having to worry about which column is indexed. You can say that in practice you don't need this, so you reduce functionality but you're no longer comparing apples to apples. Besides, it's nice to search by any column, just because relational DBs limit you doesn't mean it's not useful. map-reduce databases are something entirely different, although you can do map-reduce via joins if need be.

You denormalize because, among other things, you don't have complex joins. With column-oriented DBs that can perform joins, denormalization is not necessary. NoSql is something entirely different.