News for hackers. Stuff that matters.

A comment on Apache DataFusion. Parent comment

Re:Apache DataFusion by thebuilderjr on Sunday January 12, 2025 @09:09PM (#42679284)
I agree. The main reason I shared it is because I find it interesting as a library. I actually use it behind the scenes to build https://telemetry.sh. Essentially, I ingest JSON, infer a Parquet schema, store the data in S3 with a lookaside cache on disk, and then use DataFusion for querying.
[ Parent | on HN ]
Re:Apache DataFusion by Hugsun on Wednesday January 15, 2025 @10:09PM (#42720611)
How do you infer your Parquet schemas?
[ Parent | on HN ]
Re:Apache DataFusion by threeseed on Wednesday January 15, 2025 @11:12PM (#42721104)
You infer the types of the source data.

For example you can go through say 1% of your data and for each column see if you can coerce all of the values to a float, int, date, string etc. And then from there you can set the Parquet schema with proper types.

[ Parent | on HN ]