jagsr123

Good questions. One answer - Speed in analytics when working with large data sets matters. A lot. Think about this - several vendors seem to be claiming support for interactive analytics. i.e. i can ask an adhoc question on large volume of data and get some sort of insight in "interactive" times. Really? maybe with a thousand cores? In a competitive industry, say like in investment banking, if one can discern a pattern before the competition it simply provides an edge. Ask the question to folks involved in detecting fraud, or online portals trying to place an appropriate ad, or manufacturing plant trying to anticipate/predict faults. It isn't so much about trying to prove we are better than Spark (well other than grabbing some attention :-) ) but rather the potential to live in a world where batch processing is a thing of the past - like working with mainframes. The hope is that we can gain insight instantly. fwiw, we love Spark and expect some of these optimizations simply become part of Apache Spark.

Yes, it is a hash join.

From what I know, this test originally written by Databricks (expanded here) is meant to tease out the optimizations in the Tungsten engine. Of course, a distributed query that is dominated by shuffle costs will produce a very different result.

Just to be clear, we do support Snappydata as a library (unfortunately, the docs don't make this clear) that can work with your distribution (upto 1.6 today). Our releases will support the latest Spark version in a staggered manner.

Everything we have done is an extension to Spark. None of the existing functionality is lost. In a sense, we have turned Spark to work more like a database. So, don't think we can turn it back into the "mainline" (assuming that is what you meant).

Structured streaming really makes spark much more competitive for streaming use cases by allowing streams to be seen as DataFrames and hence make it super simple to run continuous queries using SQL on streams, but, often you also need to work with historical data in your analytic query, continuously mutate data (which may require transactional semantics), make sure the data itself is HA (not just fault tolerant; important for low latency apps), etc. What we do is fuse in modern in-memory DB with spark (i.e. DB cluster nodes run spark executors) so there is no need to couple spark with some other data management cluster. That said, some of the APIs we introduced for SQL stream processing will be replaced by the new Structured streaming APIs.

Yes. Hopefully soon.

@superkk Ironically, we debated this extensively and concluded the reverse (primarily because we believe the Spark API is succinct and integration with the eco-system will be easy) - much more like spark than Gemfire. In fact, the API is pretty much Spark. For instance, you don't care about updating data, you just use the Spark DataFrame API. ( https://github.com/SnappyDataInc/snappydata#quick-start-with... )

Metabase looks very interesting. We will go through it in more detail. Can you point to any doc on what it takes to create a connector? Note that Snappydata has two straightforward entry points - SQL (e.g. JDBC connector) or using the Spark APIs. Looks like the SQL connector route is the simplest?