News for hackers. Stuff that matters.
Re:FastSpark: A New Fast Native Implementation of Spark from... by aloknnikhil on Thursday October 31, 2019 @02:33PM (#21412088)
Spark is specifically useful for querying streaming data. How would a distributed database help with that? You'd have to build your own stream executor on top of that.
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Friday November 01, 2019 @12:59AM (#21416388)
Agreed. at the same time, Building stream execution pipeline is not rocket science. I am not saying modern distributed SQL Databases are exact replacement or clones of Spark. I am saying with little more help from the application server they are much more capable than Spark.

You can use the following options individually or in combination.

Option 1 : Pipeline DB extension (PostgreSQL)

Option 2 : Service broker in commercial SQL databases or building PUSH/PULL queue if not supported. There are many libraries in each programming language which tries to do that. Also see option 4.

Option 3 : Using CDC or Replication for synchronous or asynchronous streamed computation on single or multi node cluster

Option 4 : Transducers. For example, you can compose many sql functions or procedures to act on a single chunk of data instead of always doing async streamed computation after each stage of transformation.

Re:FastSpark: A New Fast Native Implementation of Spark from... by dominotw on Thursday October 31, 2019 @04:29PM (#21413384)
isn't all data streaming data? whats so specail about streaming data.