richieartoul
richieartoul on Hacker News
Nice article. I agree that it is a bit of a shame that everything is forced to be so S3-centric (and I say that as someone whose work helped motivate a lot of people to do that), but right now its an unfortunate reality of running software in the cloud because cloud networking and SSDs are so expensive that you really are required to use S3 if you want a system that can handle "big data" scale workloads cost effectively
Posted by
richieartoul on Sunday December 28, 2025 @03:25PM
from the show-hn dept.
Posted by
richieartoul on Monday October 06, 2025 @09:21AM
from the warpstream.com dept.
It's not a landing page, it's a blog, and if you read the first few sentences of the post it becomes immediately clear what service PlanetScale provides.
Posted by
richieartoul on Friday April 04, 2025 @11:59AM
from the warpstream.com dept.
The paper says: “in companion DNA-seq data from this project, no detectable Obelisk reads are found” which I think is getting at what you’re asking, but I’m not sure if my understanding is correct.
I’m also not sure if DNA-seq data refers to the human host, or just all DNA they were able to sequence (which would include bacteria as well I guess?)
Posted by
richieartoul on Tuesday November 19, 2024 @01:15PM
from the warpstream.com dept.
Posted by
richieartoul on Wednesday August 14, 2024 @12:26PM
from the warpstream.com dept.
Posted by
richieartoul on Tuesday July 16, 2024 @02:00PM
from the warpstream.com dept.
Posted by
richieartoul on Tuesday May 28, 2024 @01:37PM
from the warpstream.com dept.
(WarpStream co-founder here)
You don't have to keep the data stored in S3 express one zone forever, you can just land it there and then immediately compact it to S3 standard. You still pay the higher fee to write to S3EOZ, but not the higher storage fee.
WarpStream does this, data gets compacted out within seconds usually. Of course this is now... tiered storage. But implemented over two "infinitely scalable" remote storage systems so it gets rid of all the operational and scaling problems you have with a typical tiered storage Kafka setup that uses local volumes as the landing zone.
Is there a word for things that are both hilarious and tragic? I laughed out loud multiple times. Kudos to the author for making such a depressing topic so hysterical.
Posted by
richieartoul on Friday April 19, 2024 @04:52PM
from the warpstream.com dept.
Storing the data in the customer's S3 bucket is a really good idea IMO. Means they just have to trust you to run the coordination reliably, but not necessarily to keep their data secure which is a much easier bar to clear.
I think I just followed the official recommendations I found (which are probably stale now). I'll update it to r5, but it doesn't really matter. The price difference between the two is like 5%, but hardware only ends up representing a tiny fraction of Kafka's cost at scale (the real cost comes from EBS and inter-zone networking).
I could make the hardware free for Kafka in the comparison, and WarpStream would still come out significantly more cost effective. Cloud networking is really expensive.
Makes sense, very cool. Thanks!
This is pretty cool. I'm a bit of a noob when it comes to stuff like FUSE. There is a bunch of commentary in the blog post about how when DuckDB does X, the Differential Storage implementation does Y. If the differential storage implementation just exposes itself as a filesystem though, how does it know what DuckDB is doing at the application layer? For example, how does it know from the filesystem layer that DuckDB is performing a snapshot?
Do you have any more details you can share about the performance of EFS? I've never met anyone who has actually used it in anger.
Posted by
richieartoul on Monday November 20, 2023 @11:51AM
from the warpstream.com dept.
This is awesome and one of the things I’m really excited about with WASM, and specifically Wazero. The Wazero team is top notch. Now someone just needs to do this with zstd and make it fast…
Posted by
richieartoul on Wednesday October 11, 2023 @12:08PM
from the warpstream.com dept.
Posted by
richieartoul on Tuesday September 19, 2023 @12:25PM
from the warpstream.com dept.
It runs fine in AWS, Snowflake and many others run it there. The most recent FoundationDB paper goes into a lot more detail on their recovery protocol, it’s a lot more nuanced than you think, but it works extremely well
Newer versions are moving towards a custom btree storage engine called Redwood