rohankhameshra

Sure! Didn't knew there is a thread for these, but honestly from my end this was supposed to be majorly a discussion rather than hiring post, as we are still figuring out the JD and requirements, so just wanted to understand how should we go about it.

Hey HN community, Rohan this end, I am one of the founders of Datazip the company behind OLake. We have building OLake for more than a few years, and you may have seen it here couple times. In one line, OLake is a open source tool which replicates Databases (Postgres, MySQL, MSSQL, MongoDB, oracle and DB2), S3 and Kafka (it does support upserts for Kafka too).

We built it because we faced this problem first hand, stitching up 3 tools - Debezium, Kafka and Spark streaming just to get the data in Iceberg was not exactly easy and needed an entire team of 2 just to make sure its running smoothly over things like schema changes, databases restarts or just infra failures.

It writes through Glue, Hive, JDBC or REST catalogs, without Spark or a Kafka cluster in between, and has a web UI you start with docker compose or use Helm for scale.

The repo link is attached with the title. Happy to answer anything, especially if its about your experience working similar workloads!

I am a founder of a company developing commercial open source, though I agree with most things one thing which I would add is that a lot of times it is about the industry and market the product is in, for example in Data engineering ecosystem, OSS is quite successful but maybe not so much in Martech

I agree that Jev is pretty cool but example is pretty bad, there is lot more that can be done with it, particularly in unstructured data processing pipeline. One big problem I see is just figuring out if there is new training point is to be found in the data, because with stream of data coming in its waste of resource to process everything

Hi HN,

OLake just added Kafka as a source, allowing data from Kafka topics to be written directly into Apache Iceberg tables (open source, no proprietary format).

Why this was added:

Many teams today land Kafka data into warehouses or custom storage layers, then later rewrite it into Iceberg for analytics or AI workloads.

That adds latency, cost, and operational complexity.

With OLake:

- Kafka -> Iceberg is a single step

- Tables are standard Iceberg (queryable by Spark, Trino, Presto, Athena, etc.)

- Supports schema evolution and high-throughput ingestion

This is early and we’re actively looking for feedback from teams running Kafka at scale or experimenting with Iceberg-based lakehouses.

Happy to answer questions or discuss trade-offs.

As a open source contributor, agree to most of the points mentioned.

Hi everyone, I’m one of the founders of OLake. We’ve been working on a high-throughput, open-source ingestion path for Apache Iceberg, and I wanted to share the latest benchmark results and the architectural changes behind them. Here are the key numbers from the benchmark run:

- On a 4.01 billion-row dataset, OLake sustained around 319,562 rows/sec (full-load) from Postgres into Iceberg.

- The next-best ingestion tool we tested on the same dataset managed about 46,000 rows/sec, making OLake roughly 6.8× faster for full loads.

- For CDC workloads, OLake ingested change batches at around 41,390 rows/sec, compared to ~26,900 rows/sec for the closest alternative.

- Average memory usage was about 44 GB, peaking at ~59 GB on a 64-vCPU / 128 GB RAM VM.

- Parquet file output stabilized at ~300–400 MB per file (after compression), improving performance downstream and avoiding “small file” fragmentation.

How we got these improvements:

1. Rewrote the writer architecture: Parsing, schema evolution, buffering, and batch management now happen in Go. Only the final Parquet and Iceberg write path uses Java. This cut down huge amounts of serialization and JVM churn.

2. Introduced a new batching and buffering model: Instead of producing many small Parquet files, we buffer data in memory per thread and commit large chunks (roughly 4 GB before compression). This keeps throughput high and files uniform.

3. Optimized Iceberg metadata operations: Commits remain atomic even with large batches, and schema evolution happens fully in Go before any write, reducing cross-system coordination.

4. Improved operational stability: CPU, memory, and disk behaviour remained predictable even at multi-billion-row scales.

Benchmark setup:

- Dataset: ~4.01 billion rows from the NYC Taxi + FHV trips parquet sets (row width ~120–144 bytes).

- Test machine: Azure Standard D64ls v5 (64 vCPUs, 128 GB RAM).

Iceberg stored on local NVMe for the benchmark, same architecture works with S3/GCS/HDFS.

The full benchmark results, methodology, and configs are here: https://olake.io/docs/benchmarks/

And the deep-dive into how we got the ~7× speedup is here: https://olake.io/blog/how-olake-becomes-7x-faster/

I’d love feedback from the HN community, specifically around tuning (batch sizes, commit frequency, partitioning strategies), Iceberg best practices, and real-world constraints you’ve seen in high-volume pipelines.

Happy to answer questions or share configs. Thanks for taking a look!

Love Go and Rust depending on usecase, but yet to check Zig

Interesting, that will bring a big production house capabilities within Netflix itself