News for hackers. Stuff that matters.
Re:Ask HN: What do you use to open large CSV files? by chudi on Thursday August 03, 2017 @09:41AM (#14919453)
We use pandas for opening big data files, manipulating the data and then save it to a pickle file or a sql database for use consumption.
Re:Ask HN: What do you use to open large CSV files? by gaius on Thursday August 03, 2017 @04:39PM (#14923649)
DataFrame.to_sql() seems painfully slow, do you have any tips for speeding it up?

What I do is dump it as CSV again after manipulation and then bulk load it into the DB using the DB's native tool. It's messy but 10x faster.