ryanworl

Does this trick preclude the ability to sort your data within a partition? You wouldn’t be able to rely on the row IDs being sequential anymore to be able to just refer to a prefix of them within a newly created file.

This is a well-known class of optimization and the literature term is “late materialization”. It is a large set of strategies including this one. Late materialization is about as old as column stores themselves.

The equivalent to this feature is one of my favorite parts of Husky, Datadog’s storage and query system for event data.

https://youtu.be/mNneCaZewTg?si=N68fsBlYS3tuvLe3 begins at 34:32

We're still drafting our next post in this series, but the answer is actually very simple: two tiers of object storage do not have the same drawbacks as a combination of object storage and local disk. We wanted to explain that in this post too, but it would've been unreasonably long.

We've designed WarpStream to work extremely well on the slower, harder-to-use one first, and that is how 95+% of our workloads run in production. The tiered storage solutions from other streaming vendors do the opposite, where they were first designed for local SSDs and then bolted on object storage later.

The equivalent would be if we were pitching our support for an even slower, cheaper tier of object storage like AWS S3 Glacier.

This strategy will not work well for Apache Kafka because it is extremely IOPS hungry if you have more than a few partitions, and a replay of a large topic will require lots of IO bandwidth. It would work well e.g. a columnar database where a query targeting old data may only require reading a small fraction of the size of the volume, but Kafka is effectively a row-oriented storage system, so the IO pattern is different.

(WarpStream co-founder here)

We're not talking about no disks as in no storage, just nothing other than object storage. This does have a latency trade-off, but with the advent of S3 Express One Zone and Azure's equivalent high-performance tier (with GCP surely not far behind), a system designed purely around object storage can now trade cost for latency where it makes sense. WarpStream already has support for writing to a quorum of S3 Express One Zone buckets to provide regional availability, so there's not an availability trade-off here either.

Co-founder of WarpStream here. We wrote a short article explaining how the economics of this new storage class work for a system like ours: https://www.warpstream.com/blog/s3-express-is-all-you-need

WarpStream co-founder here. Implementing the Idempotent Producer feature for the Apache Kafka protocol was definitely a fun challenge for us. Please let me know if you have any questions!

[WarpStream co-founder]

WarpStream doesn't implement compacted topics today. It is on our roadmap, though. Compacted topics are typically not used in high-throughput workloads, so our plan is to delay compactions for longer than a disk-based system would to trade space amplification for write amplification.

[WarpStream CTO here]

WarpStream flushes after 4MiB of data or a configurable amount of time. Flushes can also happen concurrently.

In general, we'd prefer to not introduce many knobs. We're running a realistic throughput testing workload in our staging and production environments 24/7, so we've configured most of the knobs already to reasonable defaults.

We just added support for other S3-compatible storage systems today: https://docs.warpstream.com/warpstream/reference/use-the-age...

[WarpStream CTO here]

I'm not going to respond to your comment directly (we've already solved all the problems you've mentioned), but I thought I should mention for the sake of the other readers of this thread that you work for Redpanda which is a competitor of ours and didn't disclose that fact. Not a great look.

https://github.com/Lazin

We're aiming for per-GB usage-based pricing that is significantly cheaper than the alternatives, but the BYOC model combined with our extremely efficient cloud control plane gives us a lot of flexibility here. That's why we can offer a free tier at all.

We're mostly just not sure yet, so your input would be appreciated.

[WarpStream co-founder here]

Please sign up for our mailing list! We'll have some interesting things to announce related to this soon.

With an appropriately configured client (i.e. one that retries and waits for requests to be acknowledged), another Agent would receive the retry and the event would be written to the topic at that time.

[WarpStream co-founder here]

We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.

[WarpStream co-founder here]

My co-founder and I worked at Datadog for over 3 years where we built Husky, an event storage and query system built directly on top of S3 as well. We know what we're doing here, I promise ;)

https://www.datadoghq.com/blog/engineering/introducing-husky...

[WarpStream CTO here]

WarpStream is Kafka protocol compatible, so we do support topic-partitions and consumer groups. We do not expose support for transactions or idempotent producing today, but the internals of the system support that and we will probably work on the idempotent producer sometime in the next month, with transactions coming shortly after, depending on demand from the Developer Preview users.

(WarpStream CTO here) This is a bit subtle I will admit, but we view the Kafka protocol as a successor here because it will outlive Kafka the implementation.

[WarpStream co-founder here]

That is correct about flushing. RE: consuming. The TLDR; is that the agents in an availability zone cluster with each other to form a distributed file cache such that no matter how many consumers you attach to a topic, you will almost never pay for more than 1 GET request per 4MiB of data, per zone. Basically when a consumer fetches a block of data for a single partition, that will trigger an "over read" of up to 4MiB of data that is then cached for subsequent requests. This cache is "smart" and will deduplicate all concurrent requests for the same 4MiB blocks across all agents within an AZ.

It's a bit difficult to explain succinctly in an HN comment, but hopefully that helps.

[WarpStream co-founder here]

What do you think would be a good limit for the free plan?

This isn't actually an architectural constraint for us. We just didn't want to promise unlimited usage forever so we picked a somewhat arbitrary number to start with.

[WarpStream co-founder and CTO here]

1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our cloud control plane.

2. We think people shouldn't have to choose between reliability and costs. WarpStream gives you the reliability and availability of running in three AZs but with the cost of one.

3. We have a custom metadata database running in our cloud control plane which handles ordering.

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges.

If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

FoundationDB has never been the source of a data corruption or data loss bug in Husky to my knowledge. We would've noticed, as we noticed multiple instances of data loss in related systems or our dependencies based on our metadata in FoundationDB, and we had multiple layers of checks to reconcile the metadata in FoundationDB with the data in object storage.

I no longer work at Datadog, so my knowledge is ~3 months old.

FoundationDB is used at Datadog as the metadata store for Husky, the storage and query engine powering a significant number of Datadog products, such as logs, network performance monitoring, and trace analytics.

1. https://www.datadoghq.com/blog/engineering/introducing-husky...

2. https://www.datadoghq.com/blog/engineering/husky-deep-dive/

3. https://www.youtube.com/watch?v=mNneCaZewTg

4. https://www.youtube.com/watch?v=1-zo9jqdRZU

I was involved with this project from the beginning and it would've taken significantly longer to deliver without FoundationDB.