God, I used to work in medical billing, and people would think these file format issues were some kind of scheme. NO they are not. They deal with the obvious problem that clinicians are NOT always programmer friendly, and will put things like tabs in case notes, will use commas etc. At some point you use a delimiter that is much less likely to be used - | - is one of those, not a lot of clinicians use it and you can say, please don't use this.

A surprising amount of stuff (at least medical billing side) can be flat file moves, especially if you are billing into lots of different systems.

And yes, the idiot new person with a phD says, just quote every field with a ". Great - you program all this up, along with the required escape sequence handling, and then escapes for escapes. \ and / get used a fair bit sometimes already and we are dealing with tons of systems.

> God, I used to work in medical billing, and people would think these file format issues were some kind of scheme.

I think it must be at this point. The hot garbage that is medical data formats (HL7, in particular) and the ways you can break it as a user, unknowingly and silently. It’s incredible that a format this bad has hung around this long.

It’s not ok that users can type certain characters into a field and break the system.

Pipes are commonly used as delimiters in healthcare settings [1]. A quick quack suggests Python, Javascript and Perl ecosystems have HL7 parsing libraries available. I assume most languages do.

[1]: https://en.m.wikipedia.org/wiki/Health_Level_7

Yep, probably an HL7 message. In the case of vertical bars, pretty sure it's HL7v2

Except they usually call them “tubes”.

(Can’t find the original source, but it goes “Cant breathe? Put in a tube. Can't pee? Tube. Can't poop? Tube. Bleeding? Tube. Infection? Tube. Heart attack? Tube.”)

> But the price list is a text file delimited by pipes (vertical bars... |) which just seems unnecessary. They also do nothing to define the variables or column names they use...so that's nice.

They likely just exported the prices as HL7 from their EHR system and used Excel's built-in delimiter support, then called it a day

> HL7

And there is the problem.

Sometimes medical procedure names have commas in them. Instead of text qualifying them, it can be faster to write code that uses a different semi-standard delimiter.

It sucks, but, like, I don't really blame people for being in a hurry to fix a problem.

Or they could be using a legacy system that exports data in that format from times before CSV was invented/standardized.

For example, SCO Unix's Informix SQL's `UNLOAD TO <file> SELECT ...` queries output in a format that's very much like what avs733 describes, only that the values aren't separated by pipes, they're terminated by them, so every record ends with a pipe character.

If that's the case, there's unfortunately no built-in option to get it to include column identifiers in the report. :(

If it’s HL7 (likely) the format is more complex than just a CSV anyway. There are multiple separators for different purposes.

This is the GDPR cookie popup annoyance equivalent of requiring hospitals to disclose their chargemasters, but not spelling out that the published data should be reasonably human-readabale. Malicious compliance.

Oh but, if you read slightly between the article’s lines, you’ll notice that the law does require both human- and machine-readable publication.

The machine-readable part is mandated here: https://www.federalregister.gov/documents/2019/11/27/2019-24...

As a taste of how specific this is:

    (5) The file must use the following naming convention specified by CMS, specifically: <ein>_<hospital-name>_standardcharges.[json|xml|csv].

And here’s the consumer-readability requirement for 300 “shoppable’ services: https://www.federalregister.gov/documents/2019/11/27/2019-24...

They aren’t always quite as stupid as people make them out to be, these lawyers.

From your second link:

> c) Format. A hospital has discretion to choose a format for making public the information described in paragraph (b) of this section online.

This is how you get Excel files with certain column widths set to zero. All it has to be is searchable, free, public, and updated regularly.