Very cool, this approach will inevitably catch on, I think.

I built a similar typed PL and compiler for Morgan Stanley back in 2013: https://github.com/morganstanley/hobbes

I read about Hobbes years ago. The embeddability certainly caught my attention.

I wrote Empirical specifically for the use case of wanting to read a CSV file while inferring the type at compile time. Basically, a ton of compile-time function evaluation mixed with type providers.

Yeah, I had this type constraint system for doing that kind of thing, so I can load a file at compile time to get the type structure and then compile/evaluate to process data in the file or files. Same thing for network connections between communicating processes (type constraints at compile-time decide a protocol, machine code at run-time assumes that protocol).

Your work might predate this, but spark/scala now has good support for typed dataframes (they’re called DataSets).

Does Spark's Dataset have something like the "inferSchema" option when reading a CSV file? Everything I've read in Spark's documentation makes me think that Datasets require an explicit type provided by the user.

Empirical can infer a CSV's schema at compile time. If the file path can be determined at compile time, then the Empirical compiler will sample the CSV file and determine an appropriate type before the user's code is ever run. No need for an explicit type from the user, and yet we still have static typing.