Re:S3 as the Universal Back End
by bertgodel on Monday April 01, 2024 @06:11PM
(#39900011)
I'm still thinking about this post. S3 as a backend + the Rust/Arrow ecosystems is making unbundling data & compute a clean and straightforward reality for data engineering systems. I think this will be scary for companies that charge by storage
[ Parent
| on HN ]
I believe the main pitfall of this approach is related to what the author of the og article mentioned around r/w and query latency from S3. I like the old Twitter eng blog post on their distributed kv store called Manhattan (https://blog.x.com/engineering/en_us/a/2014/manhattan-our-re...). For any serious real-time data, we probably still need some semblance of fast R/W storage, but analytics workloads or long-running tasks can def benefit for cheap blob storage.