We use this product extensively to analyse users who play our games. We record billions of rows of data per day and can query it effectively/efficiently with Big Query. It is a pretty awesome product and getting better all the time.

That's interesting, you are the only one in this thread who is using it.

Could you tell a bit more about for what kind of analytics you are using it ? And why you use BigQuery and not some other analytics solution (Mixpanel, Kontagent, Flurry etc.) for these scenarios ?

At streak.com we use BigQuery to analyze our logs. All of our application logs are shipped over to BigQuery and we run sql commands over this large data set. Its been amazingly helpful.

We open sourced two things that helped us do this:

1) A chrome extension to make bigquery's UI better, see here: http://blog.streak.com/2012/07/streak-developer-tools-chrome...

2) A library to push logs to bigquery if you're using appengine: http://blog.streak.com/2012/07/export-your-google-app-engine...

Tangential to the discussion, but I am currently working on a system that will handle some 100 million semi-structured rows. Despite all the buzz about "big data", hadoop and nosql etc. etc. there is literally no system that can give me a way to reliably store AND search through this data on multiple columns in near real-time (assuming I can throw a reasonable no. of nodes at it).

I looked at BigQuery, but they don't support updates. It's just dump and analyze.

I needed to do something like this and I ended up settling on a custom workflow connected through zeromq dumping data into into sharded sqlite databases. Single process handled writing to each database and lots of reads were mapped out and then reduced (also via zmq). This handled around 14 billion rows when I left that job. If you want to talk some more about it, my gmail is the same as my username.

Thanks - I have emailed you.

Have you looked at hyperdex?

Yes, it's the closest to my current needs. But it's unlikely I will be able to use it currently as it's still pretty nascent and I don't feel comfortable just yet. Anyhow - I am looking to run some benchmarks on it next week.

Dumb question, how to you load the terabytes of data?

One company I know of literally ships hard drives-it's faster and more reliable than transferring the data over the internet. I'm not sure if that's what they do with BigQuery.

To get data into BigQuery you need to get it into Google Cloud Storage first and then import it from there. To get it into cloud storage you can use rest api's or command line utils. See https://developers.google.com/storage/

The associated Wired article: http://www.wired.com/wiredenterprise/2012/08/google-dremel-v...

and the ensuing HN discussion: http://news.ycombinator.com/item?id=4395164

IIRC it's been around for months now. Why is it resurfacing?

Wired.

This is a web service, do I have this right?

Wow, so Google is going to collect business and scientific raw data as well.

How would Google offer you this service if they didn't "collect" your data?

Well it won't be publicly searchable unless you decide to share it. One of the features is user-based access permissions.

But Google has access to it, which I think is vog's point.

It's not because you technically have access to something that you should or will access it, for a myriad of reasons.

> should

Sure, I guess my point is, how would they use it?

You need to read the contract/terms of service to determine what you are granting them with regards to your data. You might find that they want to keep an option open that you can't accept. You might also find that you are okay with what they propse. This is one of those times when you actually need to read the fine print.

You can do similar feats with Amazon Elastic MapReduce + Karmasphere for reasonable cost except that Amazon S3 has the annoying 5GB max file size limit, which is a pain to workaround, where Google Cloud Storage has 5TB max file size.

Isn't Amazon S3 now running with a 5 terabyte limit now[1]? But still, shipping drives is far more sane than cross the t'interwebs for things of that size.

[1] http://aws.amazon.com/s3/faqs/#How_much_data_can_I_store