The "revolutionary new tool" to replace CSVs was XML in the late 1990s.

I hate non-prettified JSON but its easy to prettify in any editor. So its a meh argument against JSON. But to solve the crap with the comma one needs a variant of JSON, and there's various of these...

One other neat feature of CSV is it can be imported in a very popular and powerful IDE, called... Excel.

The ways csv can fail is just fucking nuts. Especially when they're half hand written, half automated, or where a failure is 20m rows in. Hard to have speed and strong checks simultaneously.

The first step was just sharding the csv's into 200 lines or so with a naming schema. Then storing those in S3.

Then the following was creating a manager Lambda that would send out the schema creation task to however many Lambda concurrent instances I could run, and hold the shards until instances freed up. So it allowed me to do the inefficient work in the individual Lambda instance.

The inefficient work was determining how many uniques were in the dataset with a threshold. As in if there were 50 or more uniques, then consider that column 'dead' to becoming a schema. However, if it were less, filter it up to the main manager thread.

Then the main manager thread would binary dispatch the results to combine the results into a main schema with another unique value threshold.

Overall it was a fun learning experience. The hardest part was removing the download time to get the full file into the lambda... I never got that right. The processing was dang fast, the latency to get the rows into the individual threads never got 'fast fast'

---

Good luck! Cheers!

Hope I get back on CSV projects again soon to try your stuff out. Cheers

RFC4180 exists regardless of adoption level. In a way, the simplicity of the spec causes the proliferation of grammars. No one thinks they can just yolo a PDF by hand in a text editor. Ok, maybe PDF is a bad example. But CSV (as specified in RFC4180) is so dead simple that people take shortcuts.

pd.yeet(lambda x: yeet(x))

pd.yeet_to_csv("/clean_data/cleaned_data_1.csv")