News for hackers. Stuff that matters.
Re:A straightforward way to extend CSV with metadata by IanCal on Thursday August 19, 2021 @12:29PM (#28235877)
Excel does an atrocious job of handling CSV files. It regularly alters data, messes up encodings and either can't or couldn't (I haven't checked in a few years) open CSV files that start with a capital letter I.

Source: dealing with CSV files people exported from Excel and the horrors that flowed from there.

Re:A straightforward way to extend CSV with metadata by bitwize on Thursday August 19, 2021 @01:09PM (#28236442)
> either can't or couldn't (I haven't checked in a few years) open CSV files that start with a capital letter I.

It can't do this because it confuses such files with files in SYLK format, which was YET ANOTHER attempt to standardize spreadsheet data interchange, dating from the 80s.

Re:A straightforward way to extend CSV with metadata by IanCal on Thursday August 19, 2021 @03:14PM (#28238050)
What I find incredible about this is that it decides that the file ending ".CSV" must not be a CSV file but SYLK. Then loading it as SYLK fails, and it doesn't then try and load it as a CSV file instead.
Re:A straightforward way to extend CSV with metadata by swader999 on Thursday August 19, 2021 @12:45PM (#28236109)
I've donated hours of my life to resolving excel corrupted csv files. They haven't been satisfying hours.
Re:A straightforward way to extend CSV with metadata by IanCal on Thursday August 19, 2021 @03:22PM (#28238153)
Same. If it helps, the python package ftfy can fix a lot of encoding issues.
Re:A straightforward way to extend CSV with metadata by swader999 on Thursday August 19, 2021 @05:16PM (#28239475)
It's tempting to consider different languages...
Re:A straightforward way to extend CSV with metadata by 41209 on Thursday August 19, 2021 @01:52PM (#28237020)
Does any widely used application do it better.

I always view CSV as a lowest common denominator, of course more precise formats exist, but not everyone can use those. Csvs normally get the job done, but like anything else you need to know it's limitations. Something like a basic phone book should work, your scientific data, with dozens upon dozens of floating point numbers may not work.

Re:A straightforward way to extend CSV with metadata by IanCal on Thursday August 19, 2021 @03:21PM (#28238132)
These are not limitations of CSV files, that's excel. Not being able to open a csv file starting with the letter I has nothing to do with CSV as a format.

There just aren't that many widely used applications that deal with things like this, but Google sheets seems to be the obvious one.

Re:A straightforward way to extend CSV with metadata by capeterson on Thursday August 19, 2021 @02:57PM (#28237818)
LibreOffice is generally a little less aggressive about formatting, but still has issues similar to Excel.

For me personally, I usually view things in Excel/LO but don't save, and if I need to modify anything I'll use a text editor for one-off changes, or I'll use something like Python with the pandas library for more programmatic changes. Pandas does have issues forcing timestamps to convert sometimes, but that can be easily configured. Otherwise, no problems with this method.

There's definitely a lot of friction if you want to edit .CSV's without your formatting/encoding being altered, unfortunately.

Re:A straightforward way to extend CSV with metadata by munk-a on Thursday August 19, 2021 @06:24PM (#28240120)
I much prefer ingesting CSVs into LibreOffice since it's a lot more upfront about what encoding it's going to try and use and the delimiter it has chosen - and those choices are extremely easy to modify and flexible. Excel also has a habit of utterly clobbering character encoding on export - once you start seeing umlauts and cedillas in your text you'll notice that Excel has a really bad habit of chosen latin-1.
Re:A straightforward way to extend CSV with metadata by capeterson on Thursday August 19, 2021 @08:23PM (#28240978)
Yeah I suppose that's the crux of it. I really like that first popup that LibreOffice has when you open it up where you can specify encoding, delimiters, etc.

LibreOffice does a similar amount of clobbering, but in general it notifies you of any would-be clobbering and allows you to abort beforehand, which is really nice. I still avoid exporting via LibreOffice though for very sensitive situations, but it is noticeably better still.

Re:A straightforward way to extend CSV with metadata by pbreit on Thursday August 19, 2021 @04:48PM (#28239192)
Google Sheets handles CSVs MUCH better.