News for hackers. Stuff that matters.
Re:The Birth of Parquet by calderwoodra on Wednesday May 08, 2024 @11:57AM (#40299587)
I've spent many many hours trying these suggestions, didn't have much luck. iter_batches loads the whole file (or some very large amount of it) into memory.
Re:The Birth of Parquet by semi-extrinsic on Wednesday May 08, 2024 @04:04PM (#40302277)
It sounds like maybe your parquet file has no partitioning. Apart from the iterating over row groups like someone else suggested, I suspect there is no better solution than downloading the whole thing to your computer, partitioning it in a sane way, and uploading it again. It's only 15 GB so it should be fine even on an old laptop.

Of course then you might as well do all the processing you're interested in while the file is on your local disk, since it is probably much faster than the cloud service disk.

Re:The Birth of Parquet by okr on Saturday May 11, 2024 @06:18AM (#40327227)
What do you mean by the parquet file might have no partitioning? Is the row group size not the implicit partitioning?