How Fast Can We Write a 10M Row CSV to a Postgres Database with Pandas?

Posted by otacust on Wednesday March 02, 2022 @09:03AM
from the innerjoin.bit.io dept.

https://innerjoin.bit.io/populating-a-postgresql-table-with-pandas-is-slow-7bc63e9c88dc?gi=a7d988b433b7

( 1 comment | 3 points | on HN )

Re:How Fast Can We Write a 10M Row CSV to a Postgres Databas... by data_dan_ on Wednesday March 02, 2022 @01:22PM (#30530791)

I wrote this article after a colleague pointed out that the Pandas DataFrame.to_sql() method uses row-by-row INSERTs. There are plenty of good reasons for this, and the to_sql method works great with many different SQL database flavors, but it's not fast.

This article compares the performance of different methods for writing a Pandas DataFrame to a PostgreSQL database using the to_sql method on DataFrames ranging from 100 rows to 10,000,000 rows.

[ Parent | on HN ]