The distressing example that nags at me here is how the headline feature of Apache Spark 2.0 is that they Greenspunned a dynamic typing mechanism into it. And, in doing so, realized improvements in ergonomics, performance, and memory consumption. Understanding why and how is an instructive lesson.
The most promising typing system for arbitrary data wrangling that I've seen so far is the one that's built into a Python package called Dagster. It seems reminiscent, as best I can tell, of dependent typing. (Although I haven't actually spent any time with a dependently typed language, so that could be a misunderstanding on my part.) The actual checks are done at run time, of course, but I don't see any particular reason, aside perhaps from the complexity involved, why a similar mechanism couldn't be implemented as a static type check.
In general, type annotations are useful if you know the business requirements beforehand. Data science is exploratory, where type annotations can be a form of premature abstraction. Languages with traits (e.g. Scala) might be a decent compromise.
I’m speculating here, but as a software engineer who’s dealt with many different languages, I (personally) find types to increase my productivity, because it reduces errors at runtime. This is a huge benefit, and because I’ve spent a lot of time learning the target language’s type system, it doesn’t come at the cost of productivity, and like I said, I find it to improve my downstream productivity.
I’ve read that scientists have had significant mistakes in their findings in papers due to bugs, which I know is anecdotal, but it seems like adopting typed languages would reduce those errors and increase confidence in studies.
I wonder if they were able to recognize the downstream benefits of types, if they would shrug them off as irrelevant to their work.
Languages with algebraic data types also allow you to express state in ways that can make it easier to identify state changes, possibly making the missing increment more obvious.
Some languages will warn when variables are being used or modified in scopes where they are referenced but not used.
But, even with all this, yes bugs will still happen.
I definitely agree that if you're making a big custom model you should use a typed language. But indeed, that requires a lot of learning, so the investment has to be worth it. The problem is often that scientists don't program all day, so remembering language intricacies is hard. Dynamic languages that can have their whole syntax written on a postcard make things much easier from that pov.
However, it's possible to have a more lenient static type checker that only looks for cases where you're guaranteed to get an illegal operation at runtime. They answer the question "is this definitely wrong?"
For instance, if we have:
def f(x):
g(x)
h(x)
def g(x):
x.a()
def h(x):
x.b()
This sort of "this is clearly wrong" type checking is much less intrusive than the more common "I'm not positive this is correct" type checking.
If you're not doing whole-program analysis, then you may restrict the type search to the types imported into the transitive closure of the module and its dependencies. This makes type checking slightly more intrusive by sometimes forcing more module imports, but it's still much less intrusive than common type checkers.
If you're thinking about Erlang's Dialyzer, almost everyone I've heard talking about it says that it's difficult to understand and doesn't provide as much confidence as a regular type system would.
setattr(type(x),sys.argv[1],x.a)
setattr(type(x), 'b' if halts(sys.argv[1]) else 'c',x.a)Which in at least some cases means ignoring bugs. Pretty much every system I've worked on that used a language that didn't to type checking (or was too forgiving about it) has had runtime bugs because of type incompatibility or type coercion issues.
Indeed, but simplicity is the name of the game for exploratory programming. It's difficult to explore using languages requiring a lot of ceremony.
But then, that's when the time and funding constraints come in and prevent rewriting in more type sound languages. Which is why we end up with buggy python (if speed not an issue) or buggy C (if speed is an issue).
There are alternate paths. For example, APL tries to be terse enough that you can have the whole thing on a single screen. That's another attempt at bug prevention.
For example, I do my data wrangling in J.
Some statically-typed languages don't require a lot of ceremony.
Let me rephrase that: Is it a bug if I don't know about it? Another way to say it, would you pay money to prevent a problem you're not (yet) aware of?
Saying the bugs "never happen" is basically saying "I always write bug-free code."
I feel languages that don't have type checking (or where it's sloppy), force developers perform semi-automated testing (i.e. unit tests to catch type issues) when they could have systematic, fully-automated testing (for type issues, at least)
That’s all I’m saying, the data analyst that has some edge case “bug” that never gets executed probably shouldn’t pay the upfront overhead cost of type checking.
Type systems like Haskell add a lot of value, but you have to really know the language to make good use of it. I mean, just IO handling requires planning ahead. Small dynamic languages allow one to keep simple things simple. There's value in that too.