I had an interesting experience introducing more types to a Python script. 1500 lines long, and rotting because everyone was too scared to refactor it. (It had no unit tests because it was eminently hard to unit test).

It operated on two similar, but differently structured JSON files that described versions, deployments etc. One for libs and one for apps.

My initial goal was to use Pydantic to replace that untyped dict being used everywhere with a known type, simply to make the code easier to read.

So in my first pass, I simply added types to functions. Then I started replacing usages of dicts direct from JSON with Pydantic classes.

Which showed me that there were many functions that expected to operate on both libs and services, but some that operated only on one or the other, e.g., using dict keys that the other lib would never have.

But the real kicker was when I was approaching the end of this, thanks to the types, I could now see that no code paths for the service data actually led to it being persisted out into JSON again.

Any such path had been deleted in the past, but again, everyone was too scared to be bolder in their refactoring.

So adding types to the code made it far easier to reason about, and now I can delete about 500 lines in confidence.

That said, types aren't really needed for a 20 line Python script to "automate a common task without continually referring to the Bash docs".

IMO, a python script should not reach 1500 lines, unit tests would be much better.

I don't like typing mainly because any code with it looks ugly XD, and if I really care about typing, I would use a different language.

Now we are pushing all languages to be able the same. Instead of having a few languages excel in different domain, we want to use one language to do everything. Each of them will come with their own eco-system, and we use library for every little simple things, then we have hundreds of dependency resolution solutions, I personally don't find it appealing.

There are programming languages better than Python on something, maybe don't use Python for something it is not good at.

to be clear, I am not against having static typing as option

> IMO, a python script should not reach 1500 lines, unit tests would be much better.

My take was that Python's lack of static type checking led to the script being an unmaintainable mess because it was risky to refactor and gradually made impossible to audit, to the point where around 30% of its code was dead code that no one was able to spot.

Adding type checks was the key to fix that.

This anecdote feels like the norm in Python projects that fail to use static type checks. Without type hints, you are forced to manually check all code paths that directly and indirectly touch the component you need to change in order to reason if a minor change has catastrophic consequences. With static type checks, instead you only need to look at the type declaration or function signature to determine what you and cannot do. I don't understand how some Pythonista still try to pass this off as a feature instead of a fundamental problem.

Java is strongly typed and it does not prevent most of java programs to be seriously loaded with bugs.

Just look at how many big corp and institutions jsp form and website explode stupidly when you try to use them for simple operations...

While on the subject of static typing, is anyone aware of any good argument against it?

Static typing inherently requires you to prove things about your program, but there are many contexts where you don't fully know the properties you'd like to prove, or the type system isn't sufficiently powerful to express the properties that are actually interesting. In these contexts (live coding, prototyping, quick throwaway scripts, etc...) static typing can be more of a hindrance than an asset.

However, once you pivot into needing maintainable software with more guarantees about correctness, static typing becomes extremely important.

The problem for python is that it originated as a language for less-formal programs but cultural and labor-market forces resulted in those programs getting shipped as production software. It's not so elegant when it's being bolted on, and there's a large subset of python developers today who genuinely don't need it.

> Static typing inherently requires you to prove things about your program

Doesn't it just require you to express the interface you're already using and expecting to use?

> The problem for python is that it originated as a language for less-formal programs but cultural and labor-market forces resulted in those programs getting shipped as production software.

That's not a problem at all, as Python's type annotations and mypy clearly demonstrate.

> Doesn't it just require you to express the interface you're already using and expecting to use?

Depends on the type system. Some type systems have a lot of ceremony around types and force you into contrived abstractions. For instance, in Scala, if you wanted to have a function that can print a name of an animal or person, you could overload the function, create a "nameable" trait and have both Person and Animal implement it, or create a union type.

In typescript you could just write

sayHello(thing: {name: string}) { ... }

or a union type

sayHello(thing: Person | Animal)

or even Pick out a group of parameters of both Person and Animal

sayHello(thing: Pick<Person | Animal, "name">)

Same with Omit

Types in typescript are incredibly flexible and useful despite being discarded at compile time. I hope python goes down this route.

FWIW Scala can do structural typing like Typescript, but it has a slight performance impact due to using reflection so is generally avoided.

Compile time structural typing like approaches tend to use typeclasses and implicits. Bit more boilerplate, but no runtime hit or risk of failure if reflection access is denied by the JVM security manager.

That's one way to view it, yes. And with unsound type systems, generally, that's the workflow. Your program is a proof that the interface can be implemented. The challenge is that sometimes the interface doesn't express much that's interesting and, with more sound type systems, your program might be correct but inexpressible.

> That's not a problem at all, as Python's type annotations and mypy clearly demonstrate

Gradual/Optional typing is a great way to solve the split needs, but I was making the argument for why people might not want to use static typing in python even though it's available.

> Gradual/Optional typing is a great way to solve the split needs, but I was making the argument for why people might not want to use static typing in python even though it's available.

Forgive me for asking, but did you made any point that provides any basis for refusing to use static typing?

I mean, failing to meet an artificial and unmettable and largely subjective bar regarding academic purity of type system implementations is not a valid argument. The average pythonista does not care about convoluted type theories if all he does is pass a string or int to a function, and doesn't want his program to blow up if they pass by mistake an int instead of a string.

> Doesn't it just require you to express the interface you're already using and expecting to use?

Intuitively, yes. However there is actually the Curry-Howard correspondence that states: A type is a theorem, a value with that type is a proof that the theorem is true.

In that sense static typing always has you prove something, but when going from dynamic to static typing it's not the proof that is the problem – you already have that. The problem is figuring out what you're proving or how to express that property in the language of the types.

> However there is actually the Curry-Howard correspondence that states (...)

This is what I don't get in these silly "anti-static typing in python" rants: the all-or-nothig logic with the goalpost moved to an impossible to reach place where even academic purity, which is met nowhere at all, is deemed too loose to justify the effort.

But the whole world already manages to implement and adopt static typing. Where's the practical problem?

Meanwhile, Python does indeed have a working static typing system, which is optional and does do wonders, but for some reason static typing detractors take a militant step against it.

Why?

Is this a contrarian thing?

I couldn't tell you. Personally I'm all for static typing in Python, I'm pro static typing everywhere, really. I'll even take dependent types (most of the time).

I didn't intend to say "this is really hard, therefore let's not do it", I merely meant to provide additional background on the excerpt

> Static typing inherently requires you to prove things about your program

that you replied to. I assumed they were referring to this theoretical connection rather than a practical concern and wanted to elaborate for those previously unaware of this correspondence.

I've used a variety of languages in my career, both statically-typed and dynamically-typed[0]. I generally like both. Here's my take against static typing:

1. Static type checkers increase compile times. This is an acceptable tradeoff if they can catch bugs at compile time, but not if they still still allow bugs to occur at runtime.

2. If you're building a system which has to interact with other systems a lot, static types can be a hindrance if your static type system and their type system don't agree on fundamental things like how data should be modelled. This manifests itself as things like the Object-Relational Impedance Mismatch[1]. You would end up writing a lot (A LOT) of code to bridge the two, and in general, more code = more bugs.

3. Static type checkers can have bugs or quirky behaviour. This can lead to a lot of wasted time, and/or inelegant workarounds that makes your code less understandable (and therefore, more likely to have bugs in the future). This is one that I encountered recently: https://typelevel.org/blog/2018/05/09/product-with-serializa...

4. It's surprisingly hard to find any research that can demonstrate that static type checkers actually lead to fewer bugs.

[0] Python, Clojure for dynamically-typed languages, Typescript and Scala for statically-typed languages

[1] https://en.wikipedia.org/wiki/Object%E2%80%93relational_impe...

> 1. Static type checkers increase compile times.

It should be noted that this potential impact on compile times amount to a few seconds.

> 2. If you're building a system which has to interact with other systems a lot, static types can be a hindrance if your static type system and their type system don't agree on fundamental things like how data should be modelled.

I fail to understand how failing to comply with a contract is a type checking problem, and one which goes away if you no longer perform validations.

> 3. Static type checkers can have bugs or quirky behaviour. This can lead to a lot of wasted time, and/or inelegant workarounds that makes your code less understandable (and therefore, more likely to have bugs in the future).

I'm not sure if your example is valid, as in your blog you quite openly state that the root cause of your bug was your failure to specify the right return type. You can't really fault a system by complying with the definition you've provided.

> 4. It's surprisingly hard to find any research that can demonstrate that static type checkers actually lead to fewer bugs.

A quick search in Google Scholar for "static type bugs" popped about 4k search hits, the first of which is an article on quantifying detectable bugs on javascript which states that switching to typescript allowed 15% of public bugs to be detected.

A mix of Rich Hickey talks and my own experiences. Possibly just Rich Hickey.

a) You're going to write unit tests anyway. Type errors are not especially likely to slip through good testing, nor are they the most interesting kinds of bugs that can slip through good testing.

b) Null is a value of the wrong type, and most type systems let that one through.

c) Values have a habit of coming from outside the boundaries of the type system. Sockets, files, etc. You will always have to deal with (or reject) arbitrarily wacky inputs at runtime.

c) Static typing leads programmers to overspecify data and make programs unnecessarily brittle. Many struct/record types could be replaced with maps + runtime checking that the appropriate keys are present, while letting other keys pass through. Then our programs would not so often require modification to support new uses. (TBH I think Go's type system, of all things, takes a decent step toward this one with its implicitly satisfied interfaces).

I think many of these points are valid for a type system like Java's but they hold less ground for other languages

a. Yes tests are still needed but you need far less of them thanks to type you can focus testing on the critical logic instead of validating input and output types.

b. Solved issue in modern pl, even php considers null a value of separate type.

c. This is a good point, but with values coming from outside the system, I'm forced to do run time checking because of types which makes the whole program more less error prone.

d. This is a very valid point for most mainstream typed programming languages, row polymorphism is one technique which helps here. Every function only specifies the key's it needs to work so it can receive arbitrary records. I believe this is also possible in python by typing out dict keys and values.

Yes, I am looking forward to better type systems (and community habits/norms around those type systems) solving most of these problems. In particular, I would like to see the community gain enough faith in the safety guarantees of more advance type systems to back off of blind line coverage mandates. But today we have Go and Java's type systems, and the same coverage mandates as Python and Node.

In particular I would like to see the compiler enforce that you either return a legitimate value or raise an exception, and that the caller does not go down a path that calls for a value if an exception was raised. I hate that a reference can get 10 miles away before you discover there's no data there, and it being wrapped in Option or ? or a technically-not-null zero value doesn't mitigate that.

On your point about compiler enforcing legitimate values, the way I see it is that if u don't get the right values from the outside world you should immediately exit the program with a nice error message.

As the data moves further down into your core functions it should become more well defined and highly structured.

Basically you should parse messy data into the types you need and then you don't need to validate the data in each of your functions.

How to enforce this is a tricky problem for the compiler, maybe it's architectural pattern that can be enforced by libraries and frameworks.

> a) You're going to write unit tests anyway.

I fail to see the point of this remark.

At best, you're arguing in favour of reinventing the wheel. With. Every. Single. Call.

At worse, you're somehow arguing in favour of being far less productive by having to fill in boilerplate code to run your own hand-rolled and error prone and hard to maintain typing checks.

Where's the value in this?

> b) Null is a value of the wrong type, and most type systems let that one through.

I really see no point in this assertion. So what if "most type systems" handle a specific type in a way you don't personally like? Are you writing your software based on Nond alone? Just specify your type and let the static type checker flag an error if it catches the possibility of a type seeping through an interface. Heck, Python's type hints system already support None, Optional, and even union types. You can also double-check with unit tests and runtime type checks. The TypeScript community considers this a solved problem. Why should this be of any concern to the Python community?

> c) Values have a habit of coming from outside the boundaries of the type system. Sockets, files, etc. You will always have to deal with (or reject) arbitrarily wacky inputs at runtime.

I fail to see the relevance of this point. With static type checks you specify the interface you're expected to support. You write and test your code based on that interface, and if the language enforces it then everyone, including your dependencies, also declare a contract that they comply with.

How is that a problem? At all?

> d) Static typing leads programmers to overspecify data and make programs unnecessarily brittle.

This assertion is quite wrong. At best, you're complaining that there might be a programmer somewhere that over specified an interface. How is this a problem? At all?

Meanwhile, let's look at the real world. Even if you intentionally turn a blind eye to what's already offered by Python's type hints, just look at how TypeSript handles interfaces. With it, you can quite literally specify that a function's input/output parameter is just an object that might have a key of acertain type. Then, to move the dial away from loose and into rigour, you can use type guards to infer more specific types. Why is this supposed to be a problem ?

It's very clear that you have a rock-solid and passionately emotional commitment on this issue already. I'm sorry I mistook your question for good-faith interest in tradeoffs. I'm not going to engage deeply beyond this.

I work in statically typed languages for a living. On the whole I think this is a sensible choice for my employer. Sometimes I don't like it. Sometimes when I have a choice (personal projects, or ancillary scripts/utilities) I'll reach for untyped Python. These are the reasons.

a) The type checker is a lifeline that will save you from one kind of error in untested code. But there are lots of ways to screw up untested code. You need testing anyway. And what will be caught statically by the type checker, will be caught at runtime during an adequate test suite.

b) Nullability invalidates the guarantee that a type system is supposed to provide: that values are the right type. The boilerplate, error prone check is necessary anyway to deal with the null. Yes, there are better type systems that don't have this problem. But static typing as actually practiced today (in general, I'm not talking about Python) has this problem very often.

c) This is true only if you're a library. Your type system can't statically guarantee anything about the bytes that will arrive on a socket, which is your actual interface when you're a network service. Parsing and validation must be done at runtime.

d) Over-specification of data is endemic to the culture of static typing. In the large, people think they need to map any values they receive to an internally defined struct or class, dropping any parts of the record they don't understand. Not doing this is seen as incorrect, irresponsible, or as you put it, less rigorous. But doing it leads to a lot of coding time wasted on mapping between representations and threading new fields through all the places a record is handled. Some static type systems offer facilities to help alleviate this problem. I'd argue that to the extent you're using those facilities, you're walking back down the spectrum towards dynamic typing.

I'm designing a programming language that started with the goal of being similar to a statically typed version of Python. What I've discovered is that by far the best argument against static typing is datastructure design.

A dynamically typed language's datastructures don't need to know anything about what they store. This is typically done behind the scenes as storing pointers to the data instead of the data itself, and then the data can be dynamically cast to the appropriate type when used. The result is that a list is simply a list and feels very natural for a programmer to interact with. On the other hand, a statically typed language's datastructures have to make one of three ugly compromises.

Java uses Generics, where the type is erased, causing interactions with datastructures to act dynamically typed while everything else in the language is statically typed. While this sounds as simple as dynamically typed languages, in reality the border between the statically typed and dynamically typed parts of the code are complex with odd syntax such as "<T extends Comparable<T>>" or "<? extends T>".

C++ uses Templates, where special functions called templates have extra syntax for the caller to specify what type is used. The template function is then copy-pasted behind the scenes with its variable type replaced with the specified type. This maintains the static typing but at a high cost as it leads to strange, complex syntax and ugly, hard to read errors.

Finally, C uses manual memory allocation. This prevents data structure libraries that easily work with multiple datatypes from being able to be created. However, C uses an interesting trick to provide support for simple arrays: the size of the array is the size of the element times the number of elements, and the memory address of index i is the base pointer of the array plus i times the size of the element. Using these mathematical sleights of hand, C can appear to provide datastructure support, but in reality this support only extends to arrays of a fixed size. I'm sure there are plenty of ways around this to actual build complex datastructures in C, but in my opinion none of those ways are nearly as friendly as dynamic typing, generics, or templates.

Small comment, C# also has generics, but unlike Java doesn't do type erasure.

I remember that because of an essay on type erasure by a CS professor that researches language design. His comment is a sign your your language and compiler are well designed is the ability to do type erasure. And what's really important is to not do type erasure.

Barrier for entry for new developers.

I think JavaScript's huge success in the last decade is primarily attributed to the plug-and-play dependency ecosystem and dynamic typing

Beyond that, not much

> Barrier for entry for new developers.

What's the barrier exactly? And keep in mind typescript's growing popularity and momentum, which states quite clearly that the javascript community is more than willing to tolerate added layers of complexity just to be able to use static typing.

I think it being first to market in a browser was probably the most important aspect of it being so successful. Microsoft tried VBScript and it got some take-up , but it was ultimately too late.

It was also free. At the time, IDE's and compilers were pretty expensive in the Windows world.

I think arguments can be only made against specific type systems (implementation). If I could express my domain constraints in type system I would happily use it, but often this is not possible or it’s very hard. Then you need to decide if it’s worth the trouble. Maybe your runtime code gets complicated to conform to type systems and hinders readability? Or maybe compiler can prove very little about your code (old example with type of function for reversing a list)?

Static typing is normally about "the types are known at compile time". With modern JIT compilers, the difference between compile time and run time isn't so clear, so the question loses its meaning.

Honestly, my experience with Typescript has been less than ideal. I would rather not have static typing than have static typing jammed on top of a dynamically typed language. Compared to the type system of rust or elm, TS is hot garbage.