News for hackers. Stuff that matters.
Re:Announcing Apache Spark 1.1 by pwendell on Friday September 12, 2014 @02:39PM (#8309623)
Hey all - I'm the release manager for Spark 1.1. Happy to answer any questions about Spark or this release.
Re:Announcing Apache Spark 1.1 by hcrisp on Friday September 12, 2014 @03:47PM (#8310056)
Good news about the PySpark input format improvements. Does that also cover reading complex Parquet datatypes into SchemaRDDs with their native datatypes? When can we get a Databricks Cloud account (I'm already on the waiting list)?
Re:Announcing Apache Spark 1.1 by pwendell on Friday September 12, 2014 @04:59PM (#8310435)
Yeah, you can load Parquet data directly into SchemaRDD's in 1.1 and get the type conversion, including use of nested types. That's the long term solution for all of our storage integration is to go through the SchemaRDD API since it's a standard type description and we expect many data sources to integrate there.

Re: databricks cloud - shoot me an e-mail and I'll see if I can help. Right now demand exceeds supply for us on accounts, but I can try!

Re:Announcing Apache Spark 1.1 by ambrood on Friday September 12, 2014 @04:34PM (#8310297)
Don't the SchemaRDD already support Parquet? Although it'd be great if they supported CSVs.
Re:Announcing Apache Spark 1.1 by JoshRosen on Friday September 12, 2014 @05:21PM (#8310533)
There's work in progress to support importing CSV data as SchemaRDDs:

https://issues.apache.org/jira/browse/SPARK-2360 https://github.com/apache/spark/pull/1351

Re:Announcing Apache Spark 1.1 by century19 on Friday September 12, 2014 @04:42PM (#8310348)
Any plans to allow GraphX to work with Spark Streaming DStreams?
Re:Announcing Apache Spark 1.1 by pwendell on Friday September 12, 2014 @05:00PM (#8310443)
You can call GraphX algorithms right now from within the Streaming API, for instance compute a graph on a windowed view of data.

Online graph algorithms aren't there yet (probably what you mean). We just started adding online MLlib algorithms, so this is the main focus for now.

Re:Announcing Apache Spark 1.1 by chollida1 on Friday September 12, 2014 @03:15PM (#8309867)
Great to see this:

> This release adds significant internal changes to Spark focused on improving performance for large scale workloads.

We looked at Spark Streaming briefly when choosing which CEP engine to use. We ended up not using it as its performance wasn't on par with other offerings.

I hope the performance improvements they've done carry over to the spark streaming product. http://spark.apache.org/streaming/

We ended up using esper and a proprietary engine. The biggest problem with some other streaming utilities is

1) performance

2) you can't step time when doing back testing. ie when replaying a day of trading you'll often have signals that say something like be passive for the next 10 seconds and then go to the bid for 10 seconds and then mid point and if you still aren't done then cross the spread after an additional 10 seconds.

When backtesting you obviously don't play back in real time or it would take 6.5 hours to re simulate the day, you play back as fast as you can so you need your CEP system to step time as it goes so it properly fires your time based triggers.

I'm checking out memSql right now to see how it lets you step time whit queries.

Startup idea.... It would be nice to have one unified way of doing real time and batch processing of data. That way your real time trading engine can be back tested in the same way it runs in production. I think/hope Spark is on the way to solving this.

If anyone has any input on the best way to unify streaming vs batch event processing please let me know!!!

Re:Announcing Apache Spark 1.1 by jandrewrogers on Friday September 12, 2014 @04:38PM (#8310321)
It is quite possible to build a unified storage and execution kernel that will allow you to simultaneously and seamlessly blend (1) streaming ingest path processing, (2) online indexing/storage to disk at wire speed, and (3) fast online query processing that immediately reflects both storage and ingest path. Saturating a 10 GbE connection with this kind of workload on an ordinary server is pretty simple if the system is designed correctly. There is nothing technical that prevents processing 10 GbE streaming data concurrent with storing that data to disk and running queries across all of that data fast enough to saturate the outbound network. I have designed and implemented kernels that do exactly this.

When everything runs at wire speed and all data is always fully online, there is no meaningful distinction between "batch" and "streaming".

That said, the reason you generally do not see anything in open source that does it is that it requires a more sophisticated internal design and architecture than you typically see in any of the open source databases or big data platforms. Even a system designed to only run on a single machine (as opposed to a parallel cluster) is probably 50kLoC of dense, complex, low-level C++ just to get a basic kernel off the ground.

The main reasons you do not see a lot of startups doing this: The design of these types of storage/execution engines is rare knowledge. You have to design and implement most of your own algorithms and data structures -- few things can be farmed out to template libraries or system call uses. Relatively few developers are sufficiently skilled in C++ to successfully implement these kinds of systems with these kinds of performance envelopes. The code base for an MVP, excluding the expansive test infrastructure, is pretty huge, so you need a lot of man-hours with the above skills and expertise.

The high initial cost of building these types of systems and the difficulty of finding the necessary talent make them unattractive to investors as startups. You often have to spend $10-20M before you are even going to know if there is traction. That is a pricy bet.

Re:Announcing Apache Spark 1.1 by lsh123 on Friday September 12, 2014 @05:00PM (#8310442)
Everything above is a great summary of the CEP startup I was part of 7-8 years ago. It took us 3+ years to get the product off the ground to the stage where we started to see real sales. We've had to build custom code to handle networking, messaging, scheduling, storage, etc. to get to the required performance numbers.
Re:Announcing Apache Spark 1.1 by capkutay on Friday September 12, 2014 @03:57PM (#8310099)
Out of curiosity, did you end up going with an open source or proprietary streaming engine? Was performance your biggest deciding factor?
Re:Announcing Apache Spark 1.1 by ericfrenkiel on Friday September 12, 2014 @03:27PM (#8309936)
check out MemSQL, we have an in-memory row store for real-time transaction processing, as well as a flash-optimized column store for batch processing. I'm a big believer in the convergence/unification of storage engines that enable analysis along a time continuum.
Re:Announcing Apache Spark 1.1 by brkyvz on Friday September 12, 2014 @02:42PM (#8309644)
AWESOME!