cgivre

The Apache Drill Project Management Committee is pleased to announce the release of Drill 1.21. If you're not familiar with Drill, it is a federated, schema-free, MPP query engine that lets you query virtually any data source using standard SQL.

New features include: * New connectors for Delta Lake, Iceberg and GoogleSheets * New connector for super-efficient cross-cloud queries. * Added support for INSERT to Splunk/GoogleSheets/JDBC connectors * Support for new BI functionalities such as PIVOT/UNPIVOT * Much more...

Take a look at the article for complete release notes.

Data analysts spend so much time fighting with their data. But imagine you could just ask a question and get data. Introducing DataDistillr, an innovative new product which integrates the power of the latest GPT models along with a powerful open source query engine to do exactly that. What Google did for search, DataDistillr could do for your data.

I think that really sums it up.

The other issue I've seen is lacking documentation. I've been a Drill committer for several years and co-authored the O'Reilly book, Learning Apache Drill. I'm STILL finding undocumented functionality in Drill.

The last release (Drill 1.19.0) was a very significant release, firstly because it was the first release without MapR. Secondly, however, it had a lot of refactoring and offered some major improvements including connectivity with ElasticSearch, Splunk, Cassandra as well as readers for XML files and more.

The next Drill release will happen before the end of the year and also promises to be a big one. Some significant improvements are major rewrites of the Mongo connector, the ability to write to JDBC data sources, an Iceberg reader, Parquet updates and more.

For me the ease of use and versatility. Having tried the others, Presto, Impala, Dremio, none really work as well for my use cases. The fact that I can write a query to join data from an API with a PCAP, with zero schema prep is really incredible. I've also always appreciated that Drill uses standard SQL rather than a proprietary dialect of SQL is a huge win because it means the learning curve is muj lower.

Bottom line, a user can unzip Drill and with very minimal work, explore their data no matter where it is, with minimal config. This works as well on a laptop as a big cluster.

lol

I am!! Glad to hear you liked the book!

Drill does ;-) Drill also has a streaming Excel reader that works very well with large Excel files...

Try it out! I'd agree that if you have 1 CSV you don't need Drill. If you have more than one, is where it starts to get interesting. Also there is PyDrill which enables you to execute a Drill query and put the results directly into a pandas data frame. If you're an R person, there's Sergeant that does the same.

For what?

As the author of the above post, I thought I'd respond to the comments below. Firstly, I'm glad to see that people noticed... :-)

Firstly, to paraphrase Monty Python, "Drill's not dead yet." There are some efforts to get corporate sponsorship, but it will take time. HPE's withdrawal was expected, but disappointing none the less. With that said, we are still gearing up to release Drill 1.18 which will have a considerable amount of enhancements, including new formats (SPSS, HDF5, possibly SAS) as well as new storage plugins to enable Drill to connect to Druid as well as REST APIs.

Personally, I've always felt that Drill was marketed to the wrong audience. As everyone notes, there are many competitors in the big data analytics space: Spark, ES, Splunk, Presto, Impala to name a few. Where I see Drill as filling a rather unique niche in the market is the small-to medium size data analytics with complex data.

For instance, if you have a CSV file, you use Excel. If you have 100 CSVs you have to code. Or what if you have an Excel spreadsheet and you want to pull data from your corporate reference API, which happens to return JSON and uses OAUTH authentication? Drill is the kind of tool that can bridge this gap and allow analysts to rapidly get value out of these situations.

As an example, here's a demo of me building a COVID dashboard from a REST API and spreadsheet in about 15 min with zero data prep: https://youtu.be/oEOhFWm3D9A

Another example: incident response. Let's say you have a PCAP file, use Wireshark. What if you have 10GB of PCAP files? Most likely you'll have to code up some solution. With Drill however, you can query that w/o coding, which means that you can get the value out of this data faster.

I know there are people using Drill. I know when I demo Drill to analysts, they love it. (Ok.. I'm biased here but I would say that's an accurate representation of the response to my presentations) What Drill lost was an active developer community. I hope over the next few months, that Drill's users will step up a bit and contribute to code reviews and/or actual code contributions. I've been thinking about creating a security focused fork of Drill as well so we'll see what happens.

If you have ideas/comments/questions, you can email me at [email protected].