News for hackers. Stuff that matters.
Re:Ask HN: What do you use to open large CSV files? by uridium on Tuesday August 15, 2017 @09:41AM (#15018056)
http://www.schematiq.com - files are loaded externally from the Excel process but are transparently accessible from within Excel.
Re:Ask HN: What do you use to open large CSV files? by Lon7 on Thursday August 03, 2017 @10:21AM (#14919801)
I use the sql BULK INSERT command to put it in a table. It's simple, fast, and gives you options to generate error files containing any rows with malformed data.
Re:Ask HN: What do you use to open large CSV files? by gaius on Thursday August 03, 2017 @09:12AM (#14919202)
If you are confident that it is well-formed, bulk-load it into SQLite is the easiest method. There're methods in R and Pandas too.

Or you could load into the Excel data model instead of the grid, look in the Get & Transform feature. Then do your initial exploration there. Data model (formerly Power Pivot ) is very underrated IMHO.

Re:Ask HN: What do you use to open large CSV files? by mtmail on Thursday August 03, 2017 @09:48AM (#14919515)
Re:Ask HN: What do you use to open large CSV files? by chudi on Thursday August 03, 2017 @09:41AM (#14919453)
We use pandas for opening big data files, manipulating the data and then save it to a pickle file or a sql database for use consumption.
Re:Ask HN: What do you use to open large CSV files? by gaius on Thursday August 03, 2017 @04:39PM (#14923649)
DataFrame.to_sql() seems painfully slow, do you have any tips for speeding it up?

What I do is dump it as CSV again after manipulation and then bulk load it into the DB using the DB's native tool. It's messy but 10x faster.

Re:Ask HN: What do you use to open large CSV files? by jklein11 on Thursday August 03, 2017 @10:29AM (#14919856)
If you can do what ever you want to do by analyzing one line at a time I would recommend reading the file as a stream for processing.

If you can't do whatever you'd like one line at a time, read the file as a stream and write it to disk with some database(e.g sqlite, MySQL, Mongo). This will be slower but might be easier to play with the data.

Re:Ask HN: What do you use to open large CSV files? by CreMindES on Thursday August 03, 2017 @09:08AM (#14919174)
At first, I just look into it with less and then usually use R's data.table(fread) or some form of SQL for exploration.
Re:Ask HN: What do you use to open large CSV files? by partisan on Thursday August 03, 2017 @09:31PM (#14925647)
Targeting .NET and SQL Server, the LumenWorks CsvReader in conjunction with the SqlBulkCopy class. The performance quite good and avoids issues with permissions and file access when attempting to automate bcp commands.
Re:Ask HN: What do you use to open large CSV files? by dtnewman on Thursday August 03, 2017 @10:01AM (#14919609)
When you say "large", do you know _how_ large? Solutions might vary based on whether you are looking at a million rows or a billion rows.
Re:Ask HN: What do you use to open large CSV files? by tixocloud on Thursday August 03, 2017 @02:37PM (#14922405)
I'd say several million to several hundred million rows. Haven't seen a CSV that has a billion rows yet although I have had to consolidate multiple million row files into a single CSV.
Re:Ask HN: What do you use to open large CSV files? by johnmurch on Thursday August 03, 2017 @03:59PM (#14923216)
Re:Ask HN: What do you use to open large CSV files? by tixocloud on Friday August 04, 2017 @08:53AM (#14928289)
This website doesn't get past the corporate firewalls :)
Re:Ask HN: What do you use to open large CSV files? by limeblack on Thursday August 03, 2017 @01:36PM (#14921736)
Are you looking for something for free or open source?
Re:Ask HN: What do you use to open large CSV files? by tixocloud on Thursday August 03, 2017 @02:35PM (#14922382)
I'm fine with paid or free. Open source is fine as long as there is a pre-built package.