News for hackers. Stuff that matters.
Re:Apache DataFusion by Hugsun on Wednesday January 15, 2025 @10:09PM (#42720611)
How do you infer your Parquet schemas?
Re:Apache DataFusion by threeseed on Wednesday January 15, 2025 @11:12PM (#42721104)
You infer the types of the source data.

For example you can go through say 1% of your data and for each column see if you can coerce all of the values to a float, int, date, string etc. And then from there you can set the Parquet schema with proper types.