News for hackers. Stuff that matters.

A comment on Announcing Apache Spark 1.4. Parent comment

Re:Announcing Apache Spark 1.4 by nl on Thursday June 11, 2015 @06:08PM (#9702772)
Do this: http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-ti...

Then at the end instead of writing to HBase you can write JDBC to do the insert.

[ Parent | on HN ]
Re:Announcing Apache Spark 1.4 by threeseed on Thursday June 11, 2015 @11:09PM (#9703803)
JDBC works great if you have a large RDD that you want to persist in one go.

But if you are writing row by row you will need to implement your own batching algorithm and connection pooling to get any decent performance.

[ Parent | on HN ]
Re:Announcing Apache Spark 1.4 by gglanzani on Friday June 12, 2015 @03:08AM (#9704439)
You could always use mapPartition to open one connection per partition.
[ Parent | on HN ]
Re:Announcing Apache Spark 1.4 by sixdimensional on Thursday June 11, 2015 @06:19PM (#9702822)
Nice link. Spark jobs run Java code anyway, so why not write to JDBC with the results of processing. Makes perfect sense.
[ Parent | on HN ]