Re:Python's Type Checking Renaissance
by wk_end on Wednesday January 27, 2021 @01:15AM
(#25924903)
What are the sorts of problems you have when wrangling data that type systems don't help with? I'm not a data scientist, but I'm a type system guy, and I'd like to learn about ways that type systems are failing users. I took a very quick peak at Dagster and the Google results for "apache spark 2.0 dynamic typing" (are you talking about "Datasets"?) and nothing immediately jumped out as a clear sort of elucidation of what you're saying.
[ Parent
| on HN ]
Re:Python's Type Checking Renaissance
by snicker7 on Wednesday January 27, 2021 @01:38AM
(#25925024)
In data science, types change all the times. E.g. when dealing with tabular data, the columns (which might be used as type parameters) change frequently. In Julia, for example, see DataFrames.jl vs. TypedTables.jl for some trade-offs.
In general, type annotations are useful if you know the business requirements beforehand. Data science is exploratory, where type annotations can be a form of premature abstraction. Languages with traits (e.g. Scala) might be a decent compromise.
It sounds like the structural typing in Typescript might help a lot here - you don’t need to formally define any types for a record, the language just infers and tracks fields/types for you. I don’t know what the current state of data science is for TS though.