News for hackers. Stuff that matters.

A comment on Doing a database join with CSV files. Parent comment

Re:Doing a database join with CSV files by nojvek on Wednesday January 01, 2020 @11:15AM (#21928628)
It would still take a long time to actually send all that data to BigQuery
[ Parent | on HN ]
Re:Doing a database join with CSV files by carbocation on Wednesday January 01, 2020 @11:38AM (#21928804)
Since the OP noted that they were trying to accomplish this task on a c4.xlarge machine, the data is already "in the cloud" and so transfer should be blazingly fast.

Using the zipped CSV at [1] as a size estimate, I'd ballpark 300 million CSV rows as somewhere between 6.7-10.6 GB:

    (300000000/500000) * 11.10 = 6.7 GB
    (300000000/500000) * 17.68 = 10.6 GB
That could one minute to transfer. (Obviously if the rows contain a ton of data, these size estimates could be off, in an unbounded way.)

1 = http://eforexcel.com/wp/downloads-18-sample-csv-files-data-s...

[ Parent | on HN ]