Re:Druid: fast column-oriented distributed data store
by tschellenbach on Thursday March 31, 2016 @06:04PM
(#11400947)
Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.
Re:Druid: fast column-oriented distributed data store
by threeseed on Thursday March 31, 2016 @07:45PM
(#11401474)
Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS.
Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises.
And of course the ridiculous array of SQL engines on top of HDFS/S3/whatever else e.g. Hive, Spark SQL, Presto, Drill, SAP etc.
Re:Druid: fast column-oriented distributed data store
by fangjin on Thursday March 31, 2016 @07:51PM
(#11401496)
Druid's main value add to the data infrastructure space is around power user-facing data applications at scale. The queries it is best at are OLAP/business intelligence style queries. It isn't really designed to be a general processing tool such as Hadoop or Spark. The open source data space is very complex, and there are many different solutions targeted towards many different use cases. Druid is better than other solutions at some of these use cases, and worse than other solutions at others.
Re:Druid: fast column-oriented distributed data store
by deepanchor on Friday April 01, 2016 @12:42AM
(#11402689)
As someone who hasn't yet had the opportunity to use many of these systems, this was a great high-level overview of how the various systems fit together. Thanks for writing it!
Re:Druid: fast column-oriented distributed data store
by scott_s on Friday April 01, 2016 @10:39AM
(#11405067)
Your Wikipedia link to "stream processors" is for the wrong kind of stream processors. For a decades, the digital signal processing and graphics worlds have used the streaming abstraction to design programs and hardware. Their applications are typically expressed in pipelines, and do have to continually process data.
"Big data" stream processing is obviously related as its a dataflow programming model, but it's still very different in practice. The streaming abstraction is generally more free-form, and not realized directly in hardware. I contrast the two kinds of streaming in Section 2 of a paper from a few years ago: http://www.scott-a-s.com/files/pact2012.pdf
Re:Druid: fast column-oriented distributed data store
by joefkelley on Thursday March 31, 2016 @07:53PM
(#11401502)
If one of these databases included a really good SparkSQL connector (including great predicate push-down), I could see that being hugely beneficial to growth. Part of why people are using these systems is because you can connect in many different types of data. Make your database part of that ecosystem and you can gradually take in more of the analytics workload without the users losing the ability to also throw in some plain old text files or something and join between multiple worlds.
Re:Druid: fast column-oriented distributed data store
by joefkelley on Thursday March 31, 2016 @09:24PM
(#11401925)
Cool! Yeah I poked around the druid site and didn't find anything originally, but this looks pretty promising. It's hard to tell how full-featured it is without getting hands-on, but I see something like this making Druid much more usable in a lot of analytics environments.
Re:Druid: fast column-oriented distributed data store
by hbutani on Saturday April 02, 2016 @01:13PM
(#11412278)
We have couple of companies running Tableau on top of this. The deployment is Tableau - Spark ThriftServer(with our extension) - Druid.
We push down Slice and Dice and Star Join Queries as Druid Queries; all of Spark SQL is supported with some portions of a Query Plan being executed in Spark. We are working on supporting more Spark UDFs being pushed to Druid, performance improvements, and more coverage for Tableau. Further down we will support Star Schemas where some or all dimensions are not indexed.
Happy to discuss specific SQL support or deployment questions. Please reach out to us.
Re:Druid: fast column-oriented distributed data store
by threeseed on Thursday March 31, 2016 @09:43PM
(#11401989)
Cassandra, MongoDB and ElasticSearch all have SparkSQL DataSource drivers with predicate pushdown support. Most of the limitations in that support is from the database itself and not from the driver.
The biggest problems with SparkSQL is simply in its limited support for ANSI SQL. It's getting better with every release but not nearly quick enough.
Re:Druid: fast column-oriented distributed data store
by hbutani on Saturday April 02, 2016 @01:39PM
(#11412397)
I don't know the specific state of the DataSources you mention, but yes just providing a DataSource only is not enough; you can only push Filters and Projections to the underlying engine. You need to develop Query Rewrite Rules to rewrite Joins, Group Bys, Having, Limit etc to the underlying engine. For example our Rewrite engine for Druid is here: https://github.com/SparklineData/spark-druid-olap/blob/maste...
Can you be specific about ANSI SQL compliance requirements. Spark SQL is closing the gap on Hive SQL; both have decent support for analytical queries: Cubes/Rollups/Windowing etc. The only major gap between Spark and Hive SQL I know off is SubQuery predicates(exists/not exists).
Re:Druid: fast column-oriented distributed data store
by feral on Thursday March 31, 2016 @07:22PM
(#11401378)
Second that experience:
When I was at Optimizely, my team chose Druid for a large-scale analytics application, after a pretty extensive benchmarking. It was very impressive, though not trivial to set up.
Re:Druid: fast column-oriented distributed data store
by fangjin on Thursday March 31, 2016 @07:41PM
(#11401455)
Druid committer here. We spent a lot of time in the early days on making sure the system worked (at scale) and now we're spending more time to make it much easier to set up and manage.
Re:Druid: fast column-oriented distributed data store
by pookeh on Thursday March 31, 2016 @07:47PM
(#11401482)
It might be helpful perhaps to provide some kubernetes configuration set up (or like a presetup kubernetes running in vagrant) that has all the nodes correctly configured out of the box to easily get started with development and prototyping.
Re:Druid: fast column-oriented distributed data store
by vog on Friday April 01, 2016 @05:14AM
(#11403629)
I'm a bit surprised about forking the druid chef cookbook. From the blog post:
> we were delighted to find an existing community cookbook, chef-druid, that can configure and deploy a druid cluster. However, that community cookbook hasn’t been updated for nearly a year and does not support the latest version of druid. We have therefore forked off our own version, optimizely/chef-druid, which supports the latest version of druid.
Why didn't you create a pull request to the upstream project? That way, all people who are using the original cookbook could have directly benefit from your improvements, without having to discover the new fork.
Re:Druid: fast column-oriented distributed data store
by feral on Saturday April 02, 2016 @02:51PM
(#11412746)
I wasn't involved in that specific implementation decision, so can't say.
Speculating personally, it was probably just something that happened for speed/ease - as opposed to trying to get an apparently abandoned repo rolling again.
We did make optimizely/chef-druid open and public though, and while I'm not with Optimizely any more, I'm sure the good folk there would be more than happy to merge back into the original repo if anyone was still maintaining it and wanted that to happen.
Re:Druid: fast column-oriented distributed data store
by mau on Friday April 01, 2016 @08:53AM
(#11404398)
Much of the effort for the next release of Druid (0.9) has been put into docs. Even with that you still need to handle a complex cluster made of several services: zookeeper, druid overlord, druid middlemanager, druid broker, etc... still it sounds more complicate than it actually is. What was the alternative solution adopted by your company?
Re:Druid: fast column-oriented distributed data store
by RyanHamilton on Friday April 01, 2016 @07:06AM
(#11403986)
"becoming the leading open source solution" is something you can say that doesn't mean much and is difficult to prove or disprove. There are a number of entrenched players, open source and commercial that provide a column oriented database:
http://www.timestored.com/time-series-data/column-oriented-d...
That's a short description of 12 alternatives I wrote up. Druid does look interesting, it appears to have got the architecture pretty spot on and I'm going to look into it more.
Re:Druid: fast column-oriented distributed data store
by NamTaf on Thursday March 31, 2016 @10:24PM
(#11402131)
The realtime ingestion is interesting especially if I can still batch import. When processing machine data, I've found that a quantity of sources come in chunks (logfiles written out every 24 hours for exmaple) but the eventual aim is to migrate to realtime (i.e.: a data point every n seconds/minutes/etc. where you instantly consume that data point) streaming.
If this transition is easy without reworking infrastructure, the solution is far more attractive.
Re:Druid: fast column-oriented distributed data store
by jnordwick on Thursday March 31, 2016 @06:21PM
(#11401044)
Every open source column database I've seen is very poor: text, no decent array oriented ability (give me the prevoius row), slow, json output, etc. When will somebody get it right?
Re:Druid: fast column-oriented distributed data store
by jnordwick on Thursday March 31, 2016 @07:07PM
(#11401305)
I mean like give me the preceding row from the current. Very useful for calculating anything from a simple rate of change to a properly doing a temporal database. Arrays as a column type doesn't get you that.
Re:Druid: fast column-oriented distributed data store
by techwizrd on Thursday March 31, 2016 @11:08PM
(#11402287)
A friend of mine who interned with me at eBay used Druid and Angular to great success to build a tool for analysts to look at trends in our data. Druid is some seriously cool stuff.
Re:Druid: fast column-oriented distributed data store
by Exuma on Friday April 01, 2016 @10:11AM
(#11404854)
Our 2-man team set up Druid........ i took 5+ months and was excruciating to configure and get running smoothly (things were slightly more complicated because we decided to use docker). It also took ~30 servers to make a truly fault-tolerant setup.
With that said, it works very well, but it definitely came at the cost of a good dose of sanity.
Re:Druid: fast column-oriented distributed data store
by fangjin on Friday April 01, 2016 @11:47AM
(#11405683)
FWIW, Imply repackaged Druid in such a way that should make it much much easier to set up and evaluate. We've been porting our docs over to Druid for 0.9.0: http://imply.io/docs/latest/quickstart
Re:Druid: fast column-oriented distributed data store
by whitegrape on Friday April 01, 2016 @01:24PM
(#11406661)
Yes, but that doesn't mean you can't benchmark them anyway. And I think you could probably find some meaningful comparison. Certainly it would be more useful than the Druid whitepaper's benchmarks against MySQL. (I used to work on a DB project, we too had a benchmark against MySQL even though our DB was OLAP-focused.)
Re:Druid: fast column-oriented distributed data store
by gglanzani on Friday April 01, 2016 @06:56AM
(#11403954)
Analytics like sum(column) are much faster if data is saved column-oriented. Also every time you don't do select *, but select a, b, it's much easier on disk because you're not even touching some columns.
Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises.
And of course the ridiculous array of SQL engines on top of HDFS/S3/whatever else e.g. Hive, Spark SQL, Presto, Drill, SAP etc.
I wrote my interpretation of the current open source data landscape here for anyone interested: http://imply.io/post/2015/11/04/big-data-zoo.html
"Big data" stream processing is obviously related as its a dataflow programming model, but it's still very different in practice. The streaming abstraction is generally more free-form, and not realized directly in hardware. I contrast the two kinds of streaming in Section 2 of a paper from a few years ago: http://www.scott-a-s.com/files/pact2012.pdf
- Harish.
The biggest problems with SparkSQL is simply in its limited support for ANSI SQL. It's getting better with every release but not nearly quick enough.
Can you be specific about ANSI SQL compliance requirements. Spark SQL is closing the gap on Hive SQL; both have decent support for analytical queries: Cubes/Rollups/Windowing etc. The only major gap between Spark and Hive SQL I know off is SubQuery predicates(exists/not exists).
When I was at Optimizely, my team chose Druid for a large-scale analytics application, after a pretty extensive benchmarking. It was very impressive, though not trivial to set up.
Blog post with a bit more detail: https://medium.com/engineers-optimizely/slicing-and-dicing-d...
> we were delighted to find an existing community cookbook, chef-druid, that can configure and deploy a druid cluster. However, that community cookbook hasn’t been updated for nearly a year and does not support the latest version of druid. We have therefore forked off our own version, optimizely/chef-druid, which supports the latest version of druid.
Why didn't you create a pull request to the upstream project? That way, all people who are using the original cookbook could have directly benefit from your improvements, without having to discover the new fork.
Speculating personally, it was probably just something that happened for speed/ease - as opposed to trying to get an apparently abandoned repo rolling again.
We did make optimizely/chef-druid open and public though, and while I'm not with Optimizely any more, I'm sure the good folk there would be more than happy to merge back into the original repo if anyone was still maintaining it and wanted that to happen.
That's a short description of 12 alternatives I wrote up. Druid does look interesting, it appears to have got the architecture pretty spot on and I'm going to look into it more.
Elasticsearch and Redshift are.
Seriously: What, specifically, do you disagree with, in the context of your use case, and what is your use-case?
If this transition is easy without reworking infrastructure, the solution is far more attractive.
http://www.postgresql.org/docs/current/static/functions-wind...
With that said, it works very well, but it definitely came at the cost of a good dose of sanity.
There's also a production-ready docker distribution: https://hub.docker.com/r/imply/imply/
https://xkcd.com/927/