super_ar

Good question. I wouldn’t say this replaces Flink in general. If you already run Flink and are comfortable with it, it’s a very powerful system.

Where we saw friction with Flink was mainly: 1.) Operational overhead (jobs, state backends, checkpointing) 2.) Generic sinks not being optimized for ClickHouse (batching, small inserts, etc.)

We focused on making scaling a property of the pipeline itself (just add replicas) and optimizing specifically for ClickHouse ingestion patterns.

So Flink is more general, this is more opinionated and focused on this specific use case.

I am seeing this pattern a lot lately. Teams start with a simple flow:

logs/metrics → Vector → ClickHouse

Works well as long as they run simple transformations via Vector. When they start adding things like dedupe, longer time windows, more data volume or joins, things start to break. They actually start using Vector as a stream processing engine.

Very typical issue that I see:

Time window limits: By default vector handles windowing in-memory. So with a higher load, it becomes too heavy to run there.

Missing support: When running in prod env, I have seen teams under pressure because there is no support available (except for Datadog customers). But most people I know run it self-hosted.

Scaling hits ceiling: I keep hearing similar numbers: 250k to 300k rec/sec per instance. Even by adding more resources, things do not scale. The consequences are: backpressure, latency spikes, etc.

At that point, it is no longer a “log pipeline.” It is a streaming system. Just not treated like one.

I wrote a deeper breakdown of this here if anyone’s curious:

https://www.glassflow.dev/blog/when-vector-becomes-your-stre...

Curious how people here are handling this.

Are you still pushing more logic into Vector, or have you split it out elsewhere?

Really cool!

Looks cool! Do you have any idea who "good" it is at detecting AI-generated text?

This is interesting. Just wondering about your traffic volume and how long you have been running lcoalpdf?

For us, it is more like 5% of the traffic from GEO, but we have been running the company for 2 years and have created a lot of handwritten content for devs.

Yes, same machine.

There is another test that we published on our docs page. You can check it out here:

Setup: https://docs.glassflow.dev/load-test/setup

Results: https://docs.glassflow.dev/load-test/results

Fair point. Thanks for calling it out! To clarify, we’re focused on a specific use case: Kafka to ClickHouse pipelines with exactly-once guarantees. Kafka can’t provide exactly-once out of the box when writing to external systems like ClickHouse. You could use something like Flink, but there’s no native Flink-to-ClickHouse connector and Flink requires certain ops effort from the teams. Our goal was to show users a very easy-to-reproduce load test to validate the results. As a next step, we’re actively working on a Kubernetes-ready version that will scale horizontally and plan to share those higher-throughput results with the HN community soon.

Totally fair point. For stable, known workloads, you can get really far with something lightweight on a single machine. The challenge comes when you need fault tolerance, scaling, and delivery guarantees without constantly jumping in to fix things. Often heard from data teams talking about data peaks that they cannot predict as easily. But yes, a lot of existing tools make you pay a high-efficiency cost for that. At GlassFlow we are trying to hit that sweet spot...efficient but still resilient.

Hi HN, A few weeks ago, we shared GlassFlow: Open Source streaming ETL to dedup and join streams from Kafka for ClickHouse (https://news.ycombinator.com/item?id=43953722).

One of the top questions we received was: “How well does it perform at high throughput?”

We ran a load test and would like to share some results with you.

Summary of the test:

- Tested on 20m records

- Kafka produced 55,000 records/sec

- Processing rate of GlassFlow (deduplication): 9,000+ records/sec

- Measured on a MacBook Pro (M3 Max)

- End-to-end latency: <0.12 ms per request

Here is the blog post with full test results and tried with different parameters (rps, # of publishers, etc.): https://www.glassflow.dev/blog/load-test-glass-flow-for-clic...

It was important to us to set up the testing in a way that everybody could reproduce. Here are the docs: https://docs.glassflow.dev/load-test/setup

We would love to get feedback, especially from folks consuming high-throughput in ClickHouse.

Thanks for reading!

Ashish and Armend (founders)

Hi HN, A few weeks ago, we shared GlassFlow: Open Source streaming ETL to dedup and join streams from Kafka for ClickHouse (https://news.ycombinator.com/item?id=43953722).

One of the top questions we received was: “How well does it perform at high throughput?”

We ran a load test and would like to share some results with you.

Summary of the test:

- Tested on 20m records

- Kafka produced 55,000 records/sec

- Processing rate of GlassFlow (deduplication): 9,000+ records/sec

- Measured on a MacBook Pro (M3 Max)

- End-to-end latency: <0.12 ms per request

Here is the blog post with full test results and tried with different parameters (rps, # of publishers, etc.): https://www.glassflow.dev/blog/load-test-glass-flow-for-clic...

It was important to us to set up the testing in a way that everybody could reproduce. Here are the docs: https://docs.glassflow.dev/load-test/setup

We would love to get feedback, especially from folks consuming high-throughput in ClickHouse.

Thanks for reading!

Ashish and Armend (founders)

Great question! RMT can work well when eventual consistency is acceptable and real-time accuracy isn't critical. But in use cases where results need to be correct immediately (dashboards, alerts, monitoring, etc.), waiting on background merges doesn't work.

Here 2 more detailed examples:

Real-Time fraud detection in logistics: Let's say you are streaming events from multiple sources (payments, GPS devices, user actions) for a dashboard that should trigger alerts when anomalies happen. Now you have duplicates (retries, partial system failure, etc.). Relying on RMT means incorrect counts until merges happen. This situation can lead to missed fraud, later interventions, etc.

Event collection from multi-systems like CRM + E-commerce + Tracking: Similar user or transaction data can come from multiple systems (e.g., CRM, Shopify, internal event logs). The same action might appear in slightly different formats across streams, causing duplicates in Kafka. ClickHouse can store these, but it doesn't enforce primary keys, so you end up with misleading results until RMT resolves.

Thanks for asking those questions. Duplicates often come from how systems interact with Kafka, not from Kafka itself. For example, if a service retries sending a message after a timeout or if you collect similar data from multiple sources (like CRMs and web apps), you can end up with the same event multiple times. Kafka guarantees delivery at least once, so it doesn't remove duplicates.

ClickHouse doesn't enforce primary keys. It stores whatever you send. ReplacingMergeTree and FINAL are concepts on ClickHouse, but they are not optimal for real-time streams due to the background merging process that needs to be finished to ensure correct query results.

With GlassFlow, you clean the data streams before they hit ClickHouse, ensuring correct query results and less load for ClickHouse.

In your IoT case, a scenario I can imagine is batch replays (you might resend data already ingested). But if you're sure the data is clean and only sent once, you may not need this.

Great to hear that you are considering it for zenskar. We don't have a publicly available load test, but in internal checks it was able to handle 15k requests per second (locally on a MacBook Pro/M2 Docker). What is the load that you are expecting? Happy to connect.

Good question! RMT does deduplication, but its dependency on background merges that you can't control can lead to incorrect results in queries until the merge is complete. We wanted something that cleans the duplicates in real time. GlassFlow moves deduplication upstream, before data hits ClickHouse. If you think of it from a pipeline perspective, we believe it is easier to understand, as it is a block before ClickHouse.

It is a combination of both. We have a fantastic product designer colleague who takes care of the product, and a few friends who designed the website. I will forward your message to them. I am sure you've made their day. Thank you! :)

Thanks, Sai! Great question. The deduplication works based on the user-defined key, not the entire row. You can specify which field (e.g. a primary key like event_id) to use as the deduplication key. Within a defined time window, GlassFlow guarantees that only the first event with a given key will be forwarded to ClickHouse. Subsequent duplicates are rejected. Our idea was to keep ClickHouse as clean as possible.

It looks awesome! What are you guys using under the hood? I've lately seen a lot of companies building on top of ClickHouse.

Have seen this happening too often when my sister, brother, or wife were using the iPhone time limit for apps.