Types for Tables: A Language Design Benchmark

Re:Types for Tables: A Language Design Benchmark by chrisaycock on Saturday December 11, 2021 @06:32PM (#29525346)

I took a quick look through the example for relational extensions. I don't see anything about reading a file, much less inferring type from a file. Empirical's key innovation is that it can infer type from an external source so as long as the source can be derived at compile time.

  let path = "/path/to/files"
  let fname = "quotes.csv"

  let quotes = load(path + "/" + fname)

Since load() requires a compile-time parameter, Empirical automatically resolves the file's location ahead of time. Specifically, Empirical determines that the variables are constants and that the plus operator is pure; computing the resulting value has no side effects. Thus, the Empirical compiler accommodates load() by computing the parameter during semantic analysis. This is automatic compile-time function evaluation.

As for load(), it is a macro whose expansion includes inferring the schema of the table by sampling the first few lines of the file. This is a type provider.

Now contrast that logic against something like Apache Spark. It's written in a statically typed language (Scala), but its table semantics are dynamically typed! That's because Spark holds the schema as a runtime value and addresses column names as strings. Spark's mechanism for inferring type means it can't maintain static typing.

Empirical is the only system I know of that can infer types while staying static.

[ Parent | on HN ]

Re:Types for Tables: A Language Design Benchmark by sideeffffect on Saturday December 11, 2021 @07:28PM (#29525739)

> Empirical is the only system I know of that can infer types while staying static.

F#'s Type Providers can do that too.

https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/typ...

[ Parent | on HN ]

Re:Types for Tables: A Language Design Benchmark by chrisaycock on Saturday December 11, 2021 @08:10PM (#29526069)

F# is how I learned about type providers:

http://fsprojects.github.io/FSharp.Data/library/CsvProvider....

But look at that syntax:

  CsvProvider<"file.csv">.Load("file.csv")

The file name is listed twice! Conceivably a macro could hide that, but we'd still have the issue that the string has to be written fully. I.e., I can't just call a function to generate the string.

F# has Literals that mitigate some of this, but my understanding is that they are pretty limited:

https://github.com/fsharp/fslang-suggestions/issues/539

On the other hand, Empirical can perform automatic compile-time function evaluation on any legal expression, including across variables, user-defined functions, user-defined datatypes, etc.

  func filename(n):
    let value = String(n + 1)
    return "/path/to/" + value + ".csv"
  end

  load(filename(7))

I could put the filename in a Dataframe embedded in my source, then reference the row/column I want. Empirical's automatic CTFE handles it.

[ Parent | on HN ]

Re:Types for Tables: A Language Design Benchmark by chusk3 on Saturday December 11, 2021 @11:22PM (#29527122)

Nitpick here = the `Load` member takes a path to _any_ file of the same schema as the sample, but you can also just use the sample at runtime with the use of `Load()`. So it doesn't have to be as redundant as you've shown here, the flexibility is all about loading other instances of the same data shape.

[ Parent | on HN ]