News for hackers. Stuff that matters.

A comment on Amazon's Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2. Parent comment

Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by gregw2 on Wednesday July 31, 2024 @05:52PM (#41123862)
Are you all shifting over to storing as iceberg-enriched parquet yet and letting it (within, say Athena) manage compaction or thinking about it, or is it not worth it since this new Ray+Parquet thing is working for you?
[ Parent | on HN ]
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by thedood on Wednesday July 31, 2024 @06:09PM (#41124022)
As alluded to in the blog post, Ray+Parquet+Iceberg is the next frontier we'd like to make our compactor and similar procedures available on in open source so that the community can start bringing similar benefits for their Iceberg workloads. Stay tuned. =)
[ Parent | on HN ]