News for hackers. Stuff that matters.

A comment on Announcing Apache Spark 1.4. Parent comment

Re:Announcing Apache Spark 1.4 by threeseed on Thursday June 11, 2015 @11:09PM (#9703803)
JDBC works great if you have a large RDD that you want to persist in one go.

But if you are writing row by row you will need to implement your own batching algorithm and connection pooling to get any decent performance.

[ Parent | on HN ]
Re:Announcing Apache Spark 1.4 by gglanzani on Friday June 12, 2015 @03:08AM (#9704439)
You could always use mapPartition to open one connection per partition.
[ Parent | on HN ]