News for hackers. Stuff that matters.
Re:Google’s Dremel Makes Big Data Look Small by ryguyrg on Friday August 17, 2012 @01:23PM (#4397557)
The binary gsutil does have a multi-threaded mode for uploading many files simultaneously: https://developers.google.com/storage/docs/gsutil_reference_...

@iskander - do you have other approaches for uploading which you'd prefer?

Re:Google’s Dremel Makes Big Data Look Small by iskander on Saturday August 18, 2012 @08:03AM (#4400916)
Let me turn the question around. I have about 400GB of data (spread over ~3000 HDF files, each containing 42 compressed columns). The only way I can imagine using BigQuery on this data is if I convert it all to csv files (which will take a while) and upload them to google storage (which will also take a while). Is there any alternative?