News for hackers. Stuff that matters.
Re:ND4J: Scientific Computing on the JVM by specialist on Saturday May 18, 2019 @10:13AM (#19947471)
I had to work on some "data pipelines" built with Python & Spark. Terrible experience. All the different data structures for tabular data without a common interface. Oodles of ETL code for transmuting data structures to apply different algorithms. Foraging the docs to figure out even the inconsistent smallest tidbits.

Madness.

I am still aghast. How does anyone get any work done using those tools?

https://www.datacamp.com/community/tutorials/apache-spark-py...

Why didn't they implement RowSet interface?

Or create new tabular data holding interface?

Any way. Whenever someone enthuses about Python, Spark, big data, whatever, I now assume they're insane.

--

The whole notebook (Jupyter) idea is awesome. Are the fans conflating different tools, equating Spark with notebooks?

Re:ND4J: Scientific Computing on the JVM by mruts on Saturday May 18, 2019 @11:55AM (#19948092)
Spark is written in Scala, not Python. I've never used the Python interface, but I've extensively used Spark from Scala and while it's a little annoying to work with sometimes, for "real" Big Data (data set > 1TB maybe), its probably the best tool around.
Re:ND4J: Scientific Computing on the JVM by stochastic_monk on Saturday May 18, 2019 @06:11PM (#19950014)
Have you looked at Thrill? [0] It claims to basically be a better, C++ Spark.

[0] https://github.com/thrill