News for hackers. Stuff that matters.
Re:Announcing Apache Spark 1.4 by fleeno on Thursday June 11, 2015 @06:05PM (#9702752)
As someone who doesn't know what Apache Spark is, this article reads like it could have been randomly generated.
Re:Announcing Apache Spark 1.4 by sixdimensional on Thursday June 11, 2015 @06:14PM (#9702806)
Apache Spark is a general purpose distributed data processing and caching engine. It is an evolution of MapReduce concepts into more general "directed acylic graph" processing, which is very flexible for defining and executing data processing work on a distributed cluster. It's got some similarities to PrestoDb, Apache Drill and or Apache Storm (although not quite the same).

It also has some nice data mining libraries, a library for handling streaming data, some connectivity to external data sources and a library for accessing data stored in its generic "data frames" via SQL. "Data frames" are just an abstraction for a dataset, but they are distributed, and in-memory and/or persistent.

Personally, I like to think of as an engine for data analysis/processing and queries, but different in that it is not really a "database" like you would traditionally consider. It's almost like if you took the SQL data processing engine out of your database and made it really flexible.

Edit: Also, all the functionality of Apache Spark is programmatically accessible in Java, Scala and Python, or through SQL with their Hive/thrift interface.

Re:Announcing Apache Spark 1.4 by mdellabitta on Friday June 12, 2015 @08:43AM (#9705466)
There's an About page on that same site, available in the top navigation:

http://databricks.com/spark/about