News for hackers. Stuff that matters.

Show HN: The Future of Data APIs

Hello everyone,

I'm Matteo, and, over the last few months, I have been working with my co-founder and other folks from Goldman Sachs, Netflix, Palantir, and DBS Bank to simplify building data APIs. I have personally faced this problem myself multiple times, but, the inspiration to create a company out of it really came from this Netflix article (https://netflixtechblog.com/bulldozer-batch-data-moving-from-data-warehouse-to-online-key-value-stores-41bac13863f8).

You know the story: you have tons of data locked in your data platform and RDBMS and suddenly, a PM asks to integrate this data with your customer-facing app. Obviously, all in real-time. And the pain begins! You have to set up infrastructure to move and process the data in real-time (Kafka, Spark, Flink), provision a solid caching/serving layer, build APIs on top and, only at the end of all this, you can start integrating data with your mobile or web app! As if all this is not enough, because you are now serving data to customers, you have to put in place all the monitoring and recovery tools, just in case something goes wrong.

There must be an easier way !!!!!

That is what drove us to build Dozer. Dozer is a simple open-source Data APIs backend that allows you to source data in real-time from databases, data warehouses, files, etc., process it using SQL, store all the results in a caching layer, and automatically provide gRPC and REST APIs. Everything with just a bunch of SQL and YAML files.

In Dozer everything happens in real-time: we subscribe to CDC sources (i.e. Postgres CDC, Snowflake table streams, etc.), process all events using our Reactive SQL engine, and store the results in the cache. The advantage is that data in the serving layer is always pre-aggregated, and fresh, which helps us to guarantee constant low latency.

We are at a very early stage, but Dozer can already be downloaded from our GitHub repo (https://github.com/getdozer/dozer). We have taken the decision to build it entirely in Rust, which gives us the ridiculous performance and the beauty of a self-contained binary.

We are now working on several features like cloud deployment, blue/green deployment of caches, data actions (aka real-time triggers in Typescript/Python), a nice UI, and many others. Please try it out and let us know your feedback. We have set up a samples-repository (https://github.com/getdozer/dozer-samples) for testing it out and a Discord channel (https://discord.com/invite/3eWXBgJaEQ) in case you need help or would like to contribute ideas!

Thanks Matteo

( 3 comments | on HN )
Re:Show HN: The Future of Data APIs by gaut on Thursday April 06, 2023 @01:57PM (#35471949)
Looks really cool! Out of curiosity how did you get your team together?
Re:Show HN: The Future of Data APIs by nrjames on Friday April 07, 2023 @09:25AM (#35481335)
I've been looking for something like this because I need to build a UI on top of some relatively simple Snowflake tables and the various Retool, etc. solutions are expensive and heavy weight. I look forward to seeing the React starter example.
Re:Show HN: The Future of Data APIs by cuffe on Friday April 07, 2023 @12:25PM (#35483451)
Hi - Retool PM here. Any feedback on what you're finding heavy weight? Would love to help you get started fast