News for hackers. Stuff that matters.
Re:OpenRefine – free, open source, powerful tool for working... by Chris2048 on Monday March 23, 2020 @11:21AM (#22664931)
before you go down the spark route, consider perl/unix-tools may do this kind of thing quite well: https://livefreeordichotomize.com/2019/06/04/using_awk_and_r...
Re:OpenRefine – free, open source, powerful tool for working... by thadguidry on Monday March 23, 2020 @11:33AM (#22665042)
That author did not have Spark tuned well for the use case. This is a common issue with Spark. Since OpenRefine commonly is used with Strings, we plan to optimize in many areas for that such a few mentioned here: https://databricks.com/glossary/spark-tuning But in general, there are always tradeoffs when trying to provide immediate feedback for interactions. Since OpenRefine has many interactive features, some will need to support batching and advise the user in the interface that things will take longer...do you want to send to batch? Some of the tradeoffs and ways we plan to address these are mentioned in our general OpenRefine on Spark issue here: https://github.com/OpenRefine/OpenRefine/issues/1433