Re:BigQuery vs. Redshift vs. Athena simple benchmark
by rsynnott on Monday June 19, 2017 @08:52AM
(#14586501)
For a dataset this size, though, you'd probably realistically be using ORC/Parquet rather than CSV with Athena, which would cut query times and cost dramatically. Note that the table shows that Athena is scanning all the data each time; that would not be the case if ORC or Parquet was used.
[ Parent
| on HN ]
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by mej10 on Monday June 19, 2017 @09:05AM
(#14586599)
Do you know of any good ways of outputting log data directly into Parquet format without hadoop/spark?
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by mason55 on Monday June 19, 2017 @09:40AM
(#14586810)
I'm curious about this as well. I was evaluating Athena a few months ago and was surprised that there was no good way to get my data into Parquet or ORC format without spinning up an EMR cluster and loading into a Hive table of the desired format.
My guess is that in the past there was no real use for these formats unless you already had a Hadoop cluster running. If Amazon wants these "Hadoop as a service" concepts to take off it seems like it would be wise for them to make it easier to get data onto S3 in a better format than CSV.
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by arnon on Tuesday June 20, 2017 @02:53AM
(#14592665)
Mostly what I'm hearing here is:
"You're using it wrong"
"You should start up a Hadoop cluster, create a table, load the CSV into that table, then export it as Parquet, and then load that into S3, so that Athena can scan it"
Wouldn't you be honestly just better off creating a table and loading the CSV into a columnar database directly, like on Amazon RDS (Even if it means you bring the server instance down after you're done)?
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by vamin on Saturday June 24, 2017 @10:28AM
(#14625688)
You might look into Secor: https://github.com/pinterest/secor
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by gallamine on Monday June 19, 2017 @10:14AM
(#14587099)
I was able to use Python + Arrow to convert single files into Parquet.
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by mej10 on Monday June 19, 2017 @10:37AM
(#14587282)
I will check Arrow out, thanks!
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by rsynnott on Monday June 19, 2017 @02:52PM
(#14589383)
You can use org.apache.parquet.hadoop.ParquetWriter and its superclasses. It doesn't require a Hadoop cluster. There may also be other ways.
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by marklit on Monday June 19, 2017 @02:57PM
(#14589419)
If you mean not as a cluster Spark standalone can do conversions.
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by gabriel_b on Monday June 19, 2017 @09:42AM
(#14586835)
That's correct. I will do it and try to update the article tonight. Thanks for the feedback
Re:BigQuery vs. Redshift vs. Athena simple benchmark
by alexcasalboni on Monday June 19, 2017 @08:58AM
(#14586543)
Absolutely! Efficiency and cost will change drastically by using a columnar format!