News for hackers. Stuff that matters.
Re:Hadoop World: Rethinking the Data Warehouse with Hadoop a... by physcab on Friday November 13, 2009 @08:27PM (#941218)
No I meant 100MB. It really depends on how strained your databases are. For example, we have about 6 people constantly running jobs on MySQL and running it into the ground. So if I ever wanted to do some data manipulation I have to pretty much wait until 11pm. But yes, if your situation isn't as bad, you can do a lot with MySQL.

HiveQL is less forgiving in the sense that sometimes it doesn't return what you'd expect. Sometimes when I run DISTINCTS and GROUPBY's, the output isn't what SQL on MySQL would give. For example, DISTINCTs on strings don't work very well. GROUPBY's lose a little functionality, and you have to read the documentation to figure out what you can do.

Other than that, queries in Hive are not as cheap as they are in a traditional DB. Hive offers great flexibility, but you still have to plan your query carefully because its going to take a while to execute on your large dataset. Even if you specify LIMIT, it doesn't enforce that until after all the MapReduce jobs have run.

Re:Hadoop World: Rethinking the Data Warehouse with Hadoop a... by jhammerb on Friday November 13, 2009 @11:14PM (#941394)
If you have bucketed tables, you may find the TABLESAMPLE operator useful to iteratively construct complex queries and get query results back sooner: http://wiki.apache.org/hadoop/Hive/LanguageManual/Sampling. Agreed that Hive, and databases in general, could be better in this regard.