Re:Ask HN: What do you use to open large CSV files?
by chudi on Thursday August 03, 2017 @09:41AM
(#14919453)
We use pandas for opening big data files, manipulating the data and then save it to a pickle file or a sql database for use consumption.
[ Parent
| on HN ]
Re:Ask HN: What do you use to open large CSV files?
by gaius on Thursday August 03, 2017 @04:39PM
(#14923649)
DataFrame.to_sql() seems painfully slow, do you have any tips for speeding it up?
What I do is dump it as CSV again after manipulation and then bulk load it into the DB using the DB's native tool. It's messy but 10x faster.