News for hackers. Stuff that matters.
Re:Apache DataFusion by threeseed on Wednesday January 15, 2025 @06:54PM (#42718984)
You’re right it isn’t for you.

It’s largely for companies who can’t put everything in a single database because (a) they don’t control the source schema e.g. it’s a daily export from a SaaS app, (b) the ROI is not high enough to do so and (c) it’s not in a relational format e.g. JSON, Logs, Telemetry etc.

And with the trend toward SaaS apps it’s a situation that is becoming more common.

Re:Apache DataFusion by GardenLetter27 on Thursday January 16, 2025 @03:43AM (#42722947)
Or when the data is massive - so even BigQuery would be crazy expensive.