bitsondatadev

Is there any market or industry that you would like to work in? Like the domain itself is interesting to work in?

Trino is a nice query engine solution if you want to not just run SQL on Elasticsearch but also want to be able to join data from Elasticsearch with data in other systems Trino supports. It also supports raw elasticsearch queries that are serialized back into Trino data types.

https://trino.io/docs/current/connector/elasticsearch.html

Check out Trino Summit 2022 to learn more about the federated query engine and "Federate 'em all". Talks include Lyft, Shopify, Astronomer, Starburst, and much more to be announced in the coming weeks!

Trino (formerly PrestoSQL) is a query engine that originally aimed to replace the Hive runtime and grew into a powerful federated query engine. It can connect to Snowflake, BigQuery, Hive, Oracle, Mongo, Iceberg, Elasticsearch, and many more. This enables powerful joins across your platform from one location.

In this episode of Trino Community Broadcast, we interview Ryan Blue to discuss the latest innovations in Apache Iceberg, why choose Apache Iceberg, latest Trino innovations and much more.

Disclaimer: I am a Trino contributor.

Yeah, though a lot of Fivetran customers are likely the type that would go all in on paying for a conventional data warehouse where people using open source stacks may be the ones that are using open ingestion alternatives.

We see a pretty even mix from the Trino/Starburst lens. Bigger companies like to mix and match.

Also, you can keep track of all the BQ progress here: https://github.com/trinodb/trino/issues/6867

For managing difrerent workloads, check out this blogs and this videos from Shopify, Salesforce, Goldman Sachs, and Electronic Arts, respectively:

- https://engineering.salesforce.com/how-to-etl-at-petabyte-sc... - https://shopify.engineering/faster-trino-query-execution-inf... - https://trino.io/episodes/33.html - https://www.youtube.com/watch?v=-5mlZGjt6H4

All use the Lyft "Presto but really Trino"-Gateway project to run different clusters to handle various workloads. They go into various details for how this is achieved.

https://github.com/lyft/presto-gateway

Regarding the Trino/Presto split. I recommend looking at this blog to better understand why these two communities aren't mergeing. TL;DR Presto is a Facebook-driven project that mainly considers running on the Facebook infrastructure. Trino is community-driven that works on running well with all clouds and common infasturcture in the Trino community which is why you see a higher velocity there.

https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-... https://trino.io/blog/2020/12/27/announcing-trino.html

Soon we anticipate that Trino will become the common name in the community space but we'll always love the origins of the Trino project being Presto.

Clickhouse is a realtime system where Trino is a batch-oriented system. There are tradeoffs for doing realtime vs batch.

Realtime is generally more expensive to run as you process every individual row as it comes, batch is when you can deal with minute latency and want to handle a lot of data in chunks.

Trino is also a query engine rather than a database and it connects to many different systems: https://trino.io/docs/current/connector.html

It also happens to connect to Clickhouse and it's very common that people will use Trino to query clickhouse realtime data and join it with data in big query, an object store data lake, or Snowflake: https://trino.io/docs/current/connector/clickhouse.html

Thanks for the shoutout! :)

If you want to get started with Trino, here's a repo I created to do so: https://github.com/bitsondatadev/trino-getting-started

If you want to try an SaaS Athena alternative that's backed by Trino you can check out Starburst Galaxy: https://www.starburst.io/platform/starburst-galaxy/

Full disclosure I work at Starburst.

Check out this PR. I believe we may have tackled this one but you'd need to try it out on Trino: https://github.com/trinodb/trino/pull/1415

I mean, the hive migration path is one thing. Now that Iceberg is taking over the old Hive model, data lakes are all the rage again.

The other thing I would say is that Trino and Presto are not one-trick ponies or just hive replacements. There's also the ability to query across multiple systems that is, to me, the feature that future proofs a lot of architectures. It inherently frees you up to fiddle with your data in different systems but keep the access to that system in one location.

<3

btw, if you want to know the backstory on why Presto is now called Trino, here's the article:

https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-...

Ah this makes sense. This is similar to what Elasticsearch shards do.

Why is this just near real-time? If we're talking about Flink + Kafka, isn't that considered really real-time?

Maybe near real-time is just considering ingestion and query latency?

Yup, I would say Athena is the most commonly used version of Trino/Presto out there.

Also, if anyone is reading this wondering which version Athena comes from, it's a release that was common to both Presto and Trino before the fork. However, more recently we've worked with the Athena team on getting the newer Trino features that aren't in Presto today in Athena. So it's starting to become more based on Trino.

It’s a query engine that can talk to multiple databases and run join queries across tables from multiple sources. It also runs as a faster alternative to Hive.

Starburst employee here so take my view for what it’s worth. A few points:

Starburst spun out from some contributors working on Teradata’s Presto distro in 2017.

The Presto creators left Facebook in late 2018 and worked on building the new community and fork for most of 2019. They joined Starburst around a year later.

I asked these same questions in my last company and it seems like nothing was correlated between starburst and the new fork. After talking with Martin T. (One of the creators of Presto) about it and a few other long time contributors and my concerns went away.

I ultimately started to contribute to Trino and over the years I’ve only seen a community dedicated to making a successful query engine. You can see that by the activity and growth of the project.

Feel free to remain a skeptic but the best way to know is joint the community and make your own assumptions. :)

https://trino.io/slack.html

That’s exactly the point of the post too. Sometimes business and open source align and other times they do not. Trino was created as the project and community the puts community and individuals that contribute as the owners.

It’s not bad or evil or wrong that Facebook does this…it just is.

The latter tree-no like neutrino and the website is https://trino.io lol.

Yeah, I think that was one of the reasons why Facebook enforcing the trademark ended up being a blessing in disguise. It at least made the forks clearer. Now Trino is gaining more momentum but it takes a while for the brand recognition to set in I suppose.

Hah! It may have very well looked like that given the time constraints they were working with.

Facebook has their own internal reviews for their own internal projects and forks of open source projects. I think the issue came down to Facebook adding anyone working on the project to have the ability to merge code to the open source project.

Typically if a company wants to contribute to open source, you have to create a PR and have it be reviewed and merged by the maintainers of the open source project on top of internal review. Facebook management decided to circumvent that process.

I'm curious to know if this is anyone's first time hearing about Trino?

Here's the renaming article that clarifies this...

https://trino.io/blog/2020/12/27/announcing-trino.html

Months after this consolidation, Facebook decided to create a competing community using The Linux Foundation®. As a first action, Facebook applied for a trademark on Presto®. This was a surprising, norm-breaking move because up until that point, the Presto® name had been used without constraints by commercial and non-commercial products for over 6 years. In September of 2019, Facebook established the Presto Foundation at The Linux Foundation®, and immediately began working to enforce this new trademark. We spent the better part of the last year trying to agree to terms with Facebook and The Linux Foundation that would not negatively impact the community, but unfortunately we were unable to do so. The end result is that we must now change the name in a short period of time, with little ability to minimize user disruption.