News for hackers. Stuff that matters.
Re:S3 as the Universal Back End by bertgodel on Monday April 01, 2024 @06:11PM (#39900011)
I'm still thinking about this post. S3 as a backend + the Rust/Arrow ecosystems is making unbundling data & compute a clean and straightforward reality for data engineering systems. I think this will be scary for companies that charge by storage
Re:S3 as the Universal Back End by protus on Monday April 01, 2024 @07:15PM (#39900514)
I believe the main pitfall of this approach is related to what the author of the og article mentioned around r/w and query latency from S3. I like the old Twitter eng blog post on their distributed kv store called Manhattan (https://blog.x.com/engineering/en_us/a/2014/manhattan-our-re...). For any serious real-time data, we probably still need some semblance of fast R/W storage, but analytics workloads or long-running tasks can def benefit for cheap blob storage.