wanshao
wanshao on Hacker News
Scaling Kafka clusters has always been a challenging task. Kafka uses the ISR multi-replica mechanism to ensure data persistence, which was a natural choice when the cloud was not yet mature. However, in 2024, when cloud computing is very mature, this design seems a bit outdated. When the cluster is scaled, we must move the data of the partitions. Moving partition data will affect normal reading and writing, and this process may last a long time, tens of minutes or a few hours, depending on the amount of your data. This means that often, even when the business scale has expanded to the critical point, we still dare not carry out such operations as expansion, because there is a high execution risk.
AutoMQ is fully aware of the root cause of this problem, so it has redesigned and implemented the entire storage layer of Kafka based on the cloud (we call ourselves cloud-first Kafka). Offload data persistence to cloud storage, and ensure data persistence by the multi-replica mechanism inside cloud storage. This also gives us the ability to build a more powerful self-balancing ability than Cruise Control. This blog post details how we technically achieve this, and we hope it can bring some new insights to everyone.
Posted by
wanshao on Monday May 06, 2024 @11:29PM
from the ask-hn dept.
Streaming systems, as a crucial part of data infrastructure, are widely prevalent around us. Would you directly use Apache Kafka, or would you consider other emerging streaming systems? When making technical choices for a streaming system, what are your primary concerns? Performance, cost, latency, operational complexity, or something else?
Posted by
wanshao on Monday May 06, 2024 @11:20PM
from the ask-hn dept.
In recent years, we have seen many challengers to Kafka emerge. Without exception, they all claim to be fully compatible with the Kafka API. Some challengers have even directly forked Kafka's code to innovate further. Has the Kafka API now become the de facto standard in the stream processing domain? How has it achieved this status?
Posted by
wanshao on Monday May 06, 2024 @06:36AM
from the ask-hn dept.
In early May, Confluent released a new cluster mode entirely based on S3 called the [Freight Cluster](https://www.confluent.io/blog/introducing-confluent-cloud-freight-clusters/). This is certainly not good news for Confluent's challengers.Warpstream: Most affected. Confluent has the first-mover advantage and simultaneously offers low-latency GP clusters as well as the Freight Cluster, which...
What is BufStream?Is it a new streaming system?
Posted by
wanshao on Tuesday April 30, 2024 @07:51AM
from the ask-hn dept.
A lot of the content on HN is not entirely tech-related. I'm looking to participate in some technical discussions. I've found that Lobste.rs has some good technical discussions and exchanges. Could anyone invite me, please? My email is [email protected]
I agree with your viewpoint. The crux of the matter is not whether to use tiered storage or not, but what trade-offs have been made in the specific storage architecture and what benefits have been gained. Here(
https://github.com/AutoMQ/automq?tab=readme-ov-file#-automq-...) is a qualitative comparison chart of streaming systems including kafka/confluent/redpanda/warpstream/automq. This comparison chart does not have specific numerical comparisons, but purely based on their trade-offs at the storage level, I think this will be of some use to you.
Buddy, you've hit the nail on the head. Everything is a trade-off. For a stream processing system, I believe it's entirely possible to balance cost, ease of use, and latency. AutoMQ(
https://github.com/AutoMQ/automq) is also a stream system built on top of S3. Its storage scheme introduces a very small size of EBS storage as a persistent write cache, and then asynchronously compacts the memory data to S3, taking into account latency while retaining the advantages of warpstream. Tiered-storage is just a form, how to implement it depends on you.
Examples of this are quite common. Many tech company blog pages offer tag-based searches. The articles published can be tagged either by the system or manually, allowing us to filter content we're interested in by searching for these tags.
That makes sense. Every choice has its pros and cons. Simplicity is the philosophy of HN. If complexity gets out of control, HN would cease to exist. However, perhaps some small experiments could be feasible while maintaining simplicity?
Thank you for sharing. However, I might be looking for more specific categorizations. For instance, if I'm only interested in AWS-related content, I would like to be able to search for the 'AWS' tag.
Posted by
wanshao on Monday April 29, 2024 @05:35AM
from the ask-hn dept.
I'm a software engineer, so I'm only interested in certain topics related to development. However, HN is filled with posts of various types. The current search function is not quite sufficient when I need to find a specific type of topic.
The last conversational approach is very practical because it provides a concrete example.
This is a tremendous feat. Your success should be greatly tied to the actions you've taken. You've listed a lot of experiences that contributed to this outcome. Many projects have done the same, but they haven't succeeded. What I'm curious about is, among the suggestions you've listed, which measure has contributed the most to this result?
The ban is not just about the issue of traffic fabrication. The fake story created by this influencer has occupied limited social media resources. This has prevented many more important news and events from being understood by the public.
Posted by
wanshao on Wednesday April 17, 2024 @06:09AM
from the ask-hn dept.
We can see that many well-known projects, such as Terraform and Redis, are not using true OSS licenses. As developers, do you mind contributing code to these types of projects? Why?
Thank you for your response. I will adjust my behavior to ensure that I comply with community rules and provide something truly valuable to everyone.
Thank you for your explanation and advice. I will adjust my behavior to ensure that I can comply with the rules of the HN community.
The elasticity of the public cloud mainly manifests in the user's ability to use its resources on demand. When you need more resources to support the workload, you can quickly purchase resources; when you no longer need these resources, you can release them immediately. The elasticity of the public cloud mentioned in the article mainly refers to its ability to provide resources to users on demand.
Thanks for your kind action.I want to know is the post really dead? When I look at it in my personal account's submissions, I find that the post has not been marked as dead. My account has been set to showdead=yes. If it's really as you said, does this mean that I can't see the dead tag myself?"
It's so strange, now others can see my post as well. The title of the post is: 'Why it's difficult to utilize the elasticity of the public cloud?'. When I encountered the issue, I also sent an email to the moderator explaining the situation. I wonder if it has anything to do with their intervention. Does Hacker News have a content review process?
Posted by
wanshao on Tuesday April 16, 2024 @07:10AM
from the ask-hn dept.
Posted by
wanshao on Tuesday April 16, 2024 @06:31AM
from the automq.com dept.
When Kafka deals with large volumes of data, scaling out or in requires large data replication of the partitions, which is a very heavy operation. With the architecture based on S3 using shared storage, everything becomes much simpler. In my view, the trade-off is that we additionally use EBS as a persistent write cache to work with S3, which to some extent eliminates the latency issue of accessing S3.
Posted by
wanshao on Tuesday April 09, 2024 @02:40AM
from the automq.com dept.
jpgvm is right. AutoMQ didn't modify the computation layer of Kafka, but only revamped the storage layer. This means you can use AutoMQ just like Kafka. This is very user-friendly for those who are already using Kafka and applying tools within the Kafka ecosystem.