jhammerb

Hey,

We're on iteration 2 for this course, and it's still in somewhat rough shape. If you plan to devote significant time to the lectures, I'd recommend waiting until next spring, when we'll be teaching iteration 3 online at http://ds-class.org.

Later, Jeff

Avro defines both a row-major and a column-major layout for files: https://issues.apache.org/jira/browse/AVRO-806.

For more details on Oozie, see http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3-b2-oo....

There's been some work to add indexing to Hive. See http://www.slideshare.net/NikhilDeshpande/indexed-hive and https://issues.apache.org/jira/browse/HIVE-1803, for example.

Independence from Red Hat, since Red Hat sees VMware as their biggest competitor: http://www.pcworld.com/businesscenter/article/199870/red_hat...

Facebook gave options to employees for referrals early in their history. At some point we'll switch to cash at Cloudera as well.

from my perspective, a single user interface paradigm is unlikely to cover the variety of use cases that the full suite of hadoop platform services (flume, sqoop, hdfs, mapreduce, hive, pig, oozie, hbase) provides.

at cloudera, we've built and released into open source a user interface construction kit and an environment to house these various "applications". that way we can incorporate any metaphor you'd like: file browser, spreadsheet, yahoo pipes, etc.

see http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3b2-hue and http://www.cloudera.com/blog/2010/07/developing-applications... for details.

Not true at all. They do have other lines of business, but their databases business continues to grow. See http://seekingalpha.com/article/185179-sybase-inc-q4-2009-ea..., e.g.

After participating in a few of the XLDB meetups, at which the requirements for and design of SciDB were debated and discussed, I was quite excited about this project. I was kicked off of the mailing list unceremoniously once they decided they wanted to make it a business, and I haven't been allowed back on since. They're certainly abusing the term "open source" for marketing purposes here.

The reason they don't want people looking behind the curtain is that the rhetoric far exceeds the reality.

Hive has ODBC and JDBC drivers. What SQL functionality are you missing?

Google also kept the RPC side of Protocol Buffers, which is called Stubby, as an internal project.

Everest was replaced by Hadoop + Pig, Yahoo's take on a Hive-like system. Both Hive and Pig allow you to use SQL.

I thought Microsoft did a good job of adding a procedural-like feel to SQL with SCOPE's virtual views; see http://icme2007.org/~jrzhou/pub/Scope.pdf for more.

PigLatin certainly is a nice approach to the problem as well, of course.

Just to be clear, I was never CTO of Facebook or anything close to it (thanks for that, though!). Dustin Moskovitz and Adam D'Angelo were the only two folks to hold that title that I know about.

And yeah, I don't want to get in trouble, but ask the right people about "Team America"...

Facebook certainly has the expertise to build their own servers: http://www.facebook.com/techtalks#/video/video.php?v=2085616... and they are hiring Hardware Engineers: http://www.linkedin.com/jobs?viewJob=&jobId=822679

Well, Chris Mason is focused on btrfs, not so much ext4. See, e.g., http://www.insight24.com/webcasts/content-100695_1

Excel has incorporated software from Frontline Systems, rebranded as "Solver", for many years. Most heavy Excel users use it every day. I couldn't dig through the marketing speak entirely, but this product appears to be a port of Solver to C# and not open source. Of limited interest to HN readers, I'd expect.

Okay, copy and paste from previous threads:

We're hiring here at Cloudera.

We're especially interested in web developers who have built and deployed large, extensible applications into production environments. An interest in data visualization and analysis doesn't hurt. We also have some deep distributed storage system hacking problems.

We have a strong preference for open source experience: our team (see http://cloudera.com/about) includes core contributors from the Berkeley DB, Ganglia, Lucene/Nutch, Hadoop, and MooTools projects.

We expect you to communicate ideas clearly, exhibit preternatural intellectual curiosity across a variety of domains, write quality code, and have a consistent focus on improving yourself and the team around you.

If you're interested, drop your CV and a cover letter to [email protected].

The early release program has already started and the product is in use.

In addition to the points below about Hadoop's implementation of MapReduce, Hadoop consists of many subprojects which perform complex tasks: HDFS, a reliable, petabyte-scale distributed file system; HBase, a clone of Google's BigTable; Pig and Hive, which provide higher-level syntax, columnar storage in HDFS, and a persistent metadata repository; and Zookeeper, a coordination service for distributed systems.

I'm not impressed by how REvolution's new management is approaching marketing.

If you have bucketed tables, you may find the TABLESAMPLE operator useful to iteratively construct complex queries and get query results back sooner: http://wiki.apache.org/hadoop/Hive/LanguageManual/Sampling. Agreed that Hive, and databases in general, could be better in this regard.

Digg will be hosting an Avro hackathon on Thursday, November 19. Sign up if you'd like to hack! http://avrohackathon.eventbrite.com/