Re:Google’s Dremel Makes Big Data Look Small
by iskander on Friday August 17, 2012 @03:35AM
(#4395453)
If I remember correctly, BigQuery only lets you import data via local csv files, uploaded one at a time. That makes importing data sets of relevant size quite a pain.
[ Parent
| on HN ]
Re:Google’s Dremel Makes Big Data Look Small
by bockris on Friday August 17, 2012 @06:43AM
(#4395759)
Oddly enough, I found out about this restriction just yesterday. I wanted to play around with Google's ngram dataset. Amazon hosts it for free in S3 making it a no brainer to run in EC2 or EMR.
Google doesn't have it available in Google Cloud Storage and I would burn a ton of bandwidth and time just to get it there.
http://stackoverflow.com/questions/11990839/is-there-a-way-t...
This makes it too difficult to kick the tires on BigQuery IMO. A file that will upload in a reasonable amount of time is more or less a toy dataset and if it's big enough to be a valid test, you will probably spend days trying to get it uploaded.
Re:Google’s Dremel Makes Big Data Look Small
by proppy on Friday August 17, 2012 @09:06AM
(#4396129)
There is publicdata.samples.trigrams in the public datasets.
Re:Google’s Dremel Makes Big Data Look Small
by bockris on Friday August 17, 2012 @09:18AM
(#4396171)
Yeah, I left a comment on the SO post noting that and my need for bigrams rather than trigrams.
Re:Google’s Dremel Makes Big Data Look Small
by ryguyrg on Friday August 17, 2012 @01:23PM
(#4397557)
The binary gsutil does have a multi-threaded mode for uploading many files simultaneously: https://developers.google.com/storage/docs/gsutil_reference_...
@iskander - do you have other approaches for uploading which you'd prefer?
Re:Google’s Dremel Makes Big Data Look Small
by iskander on Saturday August 18, 2012 @08:03AM
(#4400916)
Let me turn the question around. I have about 400GB of data (spread over ~3000 HDF files, each containing 42 compressed columns). The only way I can imagine using BigQuery on this data is if I convert it all to csv files (which will take a while) and upload them to google storage (which will also take a while). Is there any alternative?
Does Google charge for bandwidth from Google App Engine to this service? It might be cheaper to upload the data to GAE in a smaller form, then transform it to CSV and upload from there.
Re:Google’s Dremel Makes Big Data Look Small
by alooPotato on Friday August 17, 2012 @12:32PM
(#4397278)
Sadly you must still go through cloud storage to get data from appengine to bigquery. No savings there except maybe time.
Re:Google’s Dremel Makes Big Data Look Small
by ryguyrg on Friday August 17, 2012 @01:26PM
(#4397570)
FYI - you can technically upload directly to BigQuery and not go through cloud storage. It just tends to be a bit more error-prone in normal implementations. We're working on documenting resumable upload to handle error cases better.