Re:Ditching Excel for Python in a legacy industry
by hnracer on Wednesday December 30, 2020 @10:42PM
(#25589350)
Does R have robust CSV parsing? I remember using the default and it'd be extremely finicky about getting the header and index flags right and wouldn't typecast numeric columns properly (instead they'd end up as factors and not play nice)
[ Parent
| on HN ]
Re:Ditching Excel for Python in a legacy industry
by st1ck on Wednesday December 30, 2020 @11:58PM
(#25589730)
Python version of data.table has very fast CSV parsing (compared to Pandas), and it didn't have issues like those you mention. Even if data.table had issues with CSV parsing, you could probably use Apache Arrow to parse CSV into arrow table and then convert it to data.table (but that is probably suboptimal).
Re:Ditching Excel for Python in a legacy industry
by alexhutcheson on Thursday December 31, 2020 @03:12AM
(#25590668)
Re:Ditching Excel for Python in a legacy industry
by bostonfincs on Wednesday December 30, 2020 @10:58PM
(#25589435)
Personally have never had a problem with R csv parsing
Re:Ditching Excel for Python in a legacy industry
by disgruntledphd2 on Thursday December 31, 2020 @05:29AM
(#25591423)
It happens, but mostly because other formats don't produce usable CSV's. The biggest problem is if there are any free-entry text fields (common for customer/business name), and there isn't full quoting around these fields, base R will break.
I believe both fread and readr::read_csv do the right thing here, but the base-R perspective on data manipulation before read.csv is to use Perl (the R-core team are pretty old-school, to be fair).