News for hackers. Stuff that matters.
Re:Google’s Dremel Makes Big Data Look Small by iskander on Friday August 17, 2012 @03:35AM (#4395453)
If I remember correctly, BigQuery only lets you import data via local csv files, uploaded one at a time. That makes importing data sets of relevant size quite a pain.
Re:Google’s Dremel Makes Big Data Look Small by bockris on Friday August 17, 2012 @06:43AM (#4395759)
Oddly enough, I found out about this restriction just yesterday. I wanted to play around with Google's ngram dataset. Amazon hosts it for free in S3 making it a no brainer to run in EC2 or EMR.

Google doesn't have it available in Google Cloud Storage and I would burn a ton of bandwidth and time just to get it there.

http://stackoverflow.com/questions/11990839/is-there-a-way-t...

This makes it too difficult to kick the tires on BigQuery IMO. A file that will upload in a reasonable amount of time is more or less a toy dataset and if it's big enough to be a valid test, you will probably spend days trying to get it uploaded.

Re:Google’s Dremel Makes Big Data Look Small by proppy on Friday August 17, 2012 @09:06AM (#4396129)
There is publicdata.samples.trigrams in the public datasets.
Re:Google’s Dremel Makes Big Data Look Small by bockris on Friday August 17, 2012 @09:18AM (#4396171)
Yeah, I left a comment on the SO post noting that and my need for bigrams rather than trigrams.
Re:Google’s Dremel Makes Big Data Look Small by ryguyrg on Friday August 17, 2012 @01:23PM (#4397557)
The binary gsutil does have a multi-threaded mode for uploading many files simultaneously: https://developers.google.com/storage/docs/gsutil_reference_...

@iskander - do you have other approaches for uploading which you'd prefer?

Re:Google’s Dremel Makes Big Data Look Small by iskander on Saturday August 18, 2012 @08:03AM (#4400916)
Let me turn the question around. I have about 400GB of data (spread over ~3000 HDF files, each containing 42 compressed columns). The only way I can imagine using BigQuery on this data is if I convert it all to csv files (which will take a while) and upload them to google storage (which will also take a while). Is there any alternative?
Re:Google’s Dremel Makes Big Data Look Small by sp332 on Friday August 17, 2012 @10:15AM (#4396425)
Does Google charge for bandwidth from Google App Engine to this service? It might be cheaper to upload the data to GAE in a smaller form, then transform it to CSV and upload from there.
Re:Google’s Dremel Makes Big Data Look Small by alooPotato on Friday August 17, 2012 @12:32PM (#4397278)
Sadly you must still go through cloud storage to get data from appengine to bigquery. No savings there except maybe time.
Re:Google’s Dremel Makes Big Data Look Small by ryguyrg on Friday August 17, 2012 @01:26PM (#4397570)
FYI - you can technically upload directly to BigQuery and not go through cloud storage. It just tends to be a bit more error-prone in normal implementations. We're working on documenting resumable upload to handle error cases better.