News for hackers. Stuff that matters.
Re:Presto: Interacting with petabytes of data at Facebook by polskibus on Wednesday November 06, 2013 @02:34PM (#6684896)
1. Can you force it to do exact (not approximate counts) and aggregations? If so, what are the preconditions (size of tables?)

2. In data warehouse terms, would you fit it as a MOLAP, ROLAP or HOLAP kind of engine? I'm not sure whether all data is held in memory and whether aggregates are cached. Can you preprocess the dataset in batch mode (let's say overnight), repopulate aggregate caches for faster retrieval later on? (I mean that as something similar to SQL Server Analysis Services in MOLAP mode).

3. Can you compare it to Apache Drill ?

Re:Presto: Interacting with petabytes of data at Facebook by electrum on Wednesday November 06, 2013 @03:08PM (#6685120)
1) Presto always returns exact answers, with the exception of the approximate aggregate functions approx_distinct() and approx_percentile() (see http://prestodb.io/docs/current/functions/aggregate.html). We do not currently support COUNT(DISTINCT), though if you really need that you can use a subquery with SELECT DISTINCT.

We have a pull request for partitioned hash aggregations, but currently all the groups must fit in memory limit specified by the "task.max-memory" configuration parameter (see http://prestodb.io/docs/current/installation/deployment.html for details).

Regarding approximate queries, we are working with the author of BlinkDB (http://blinkdb.org/) to add it to Presto. BlinkDB allows very fast approximate queries with bounded errors (an important requirement for statisticians / data scientists).

2) Presto is a traditional SQL engine, so it would be ROLAP. We don't yet have any support for building cubes or other specialized structures (though full materialized view support with rewrite is on the roadmap).

The Presto query engine is actually agnostic to the data source. Data is queried via pluggable connectors. We don't currently have any caching ready for production.

There is a very alpha quality native store that we plan to use soon for query acceleration. The idea is you create a materialized view against a Hive table which loads the data into the native store. The view then gets used transparently when you query the Hive table, so the user doesn't have to rewrite their queries or even know about it. All they see is that their query is 100x faster. (All this code is there today in the open source release but needs work to productionize it.)

We have a dashboard system at Facebook that uses Presto. For large tables, users typically setup pipelines in Hive that run daily to compute summary tables, then write the dashboard queries against these summary tables. In the future, we would like to be able to handle all of this within Presto as materialized views.

3) We're excited about the Drill project. They have some interesting ideas about integrating unstructured data processing (like arbitrary JSON documents) with standard SQL. However, last I looked they were still in early development, whereas Presto is in production at Facebook and is usable today. Please also see this comment: https://news.ycombinator.com/item?id=6684785

Re:Presto: Interacting with petabytes of data at Facebook by polskibus on Wednesday November 06, 2013 @03:15PM (#6685189)
Thanks a lot for the answers, good luck on your further development. I'm a regular SQL Server user and MOLAP goodies can be very useful. From what I understand, it should be very easy to join a new data source to existing Presto warehouse. So I suppose you could "upload" additional mappings from Excel/CSV into the warehouse and do new queries live (instead of waiting for batch reprocess). This can be very beneficial to the user/analyst who often likes to define new data categorizations on-the-fly and use them to dig into data further.Great job! I'm putting this project on my must-try-shortlist right now!
Re:Presto: Interacting with petabytes of data at Facebook by spydum on Wednesday November 06, 2013 @07:57PM (#6686760)
If this is a hot button item, seriously look into microstrategy >9.2.1. You build the project against some warehouse, and can import other data sources like excel, access, or even other odbc sources. The imported data gets converted to an in memory cube which you can then combine analysis with the larger warehouse. No reprocessing on the warehouse needed.
Re:Presto: Interacting with petabytes of data at Facebook by bugger on Wednesday November 06, 2013 @04:05PM (#6685505)
How complete is the SQL dialect? Are nested queries supported? I was surprised to see that the ANTLR grammar is only ~900 lines; a quick perusal didn't suggest to me that nested queries are supported, but I'm not a guru...
Re:Presto: Interacting with petabytes of data at Facebook by cakeface on Wednesday November 06, 2013 @04:14PM (#6685547)
> It currently supports a large subset of ANSI SQL, including joins, left/right outer joins, subqueries,and most of the common aggregate and scalar functions, including approximate distinct counts (using HyperLogLog) and approximate percentiles (based on quantile digest). The main restrictions at this stage are a size limitation on the join tables and cardinality of unique keys/groups. The system also lacks the ability to write output data back to tables (currently query results are streamed to the client).
Re:Presto: Interacting with petabytes of data at Facebook by supergirl on Wednesday November 06, 2013 @06:25PM (#6686323)
2) Can't you directly read the RCFiles/etc. bypassing Hive? Why implement another storage format?
Re:Presto: Interacting with petabytes of data at Facebook by electrum on Wednesday November 06, 2013 @07:27PM (#6686640)
Presto does read Hive data directly out of HDFS, bypassing Hive/MapReduce. However, neither HDFS nor the file formats like RCFile are efficient. The entire design of HDFS makes it difficult to build a real column store. Having a native store for Presto allows us to have tight control over the exact data format and placement, enabling 10-100x performance increases over the 10x that Presto already has over Hive.

Additionally, in the long term, we want to enable Presto to be a completely standalone system that is not dependent on HDFS or the Hive metastore, while enabling next-generation features such as full transaction support, writable snapshots, tiered storage, etc.

Re:Presto: Interacting with petabytes of data at Facebook by flyovercountry on Wednesday November 06, 2013 @10:01PM (#6687187)
HDFS now supports placement groups with one use case being columnar storage.