When starting a large project, it would be better to skip Python altogether and use a proper statically-typed language.
Type hints only make sense when you’re working on an existing large Python project - just as Guido was doing at Dropbox when he added them to the language. This is because type annotations are the worst-of-both-worlds - they require the verbosity of static typing and provide few of its benefits.
> This is not a course for software engineers on how to write or maintain a one-million line Python application. I don't write programs like that, nor do most companies who use Python, and neither should you.
But otherwise, I feel like trying to choose the language that scales to a million lines sounds like dooming in the beginning itself by over-engineering. I attempted that myself but kept coming back to python because it's fast to code, it's forgiving as hell and its customizable. Once you're used to it's quirks you can breeze through it fast. I'm sure ruby and js have the same as well, and with ts you get better typing, but with the typing you can do with 3.8, I'll argue that python gives the _best_ of both worlds if it's especially mixed with a good IDE like pycharm. If you really want, you can also incorporate pytype or something from the get-go. If my Greenfield project's biggest problem is that python doesn't scale, then I'm a very happy man.
I am willing to entertain the idea that I have been very lucky or that I have some programming mannerism which has caused me to skate by this kind of thing, but I just don't get it.
At the same time, it is not impossible to write a large program entirely in assembly language, it just requires extreme discipline, even more so than python.
It's just a question of degree.
But when you DO get a type problem, and in any sizable code base, you WILL, it only manifests AT RUNTIME, and possibly at a customer site or in production, rather than manifesting at compile time.
Case in point, I had written an apartment search website in Perl. Now, at this point I had been pretty disenchanted with Perl as a whole due to the "there's more than one way to do it" culture combining with the "executable line noise" syntax to give rise to a lot of very impenetrable write-once, read-never-again code from others. So, when I did this, I used real, appropriately-named variables, eschewing the convenience of $_, and I made sure each line of code did one thing and only one thing if at all possible. I did not pack a lot into a line. Each line should be obvious. Where I felt that it might be subject to interpretation, I added comments as to what I was doing and why. Each function had is own comment section which discussed what it was for and why, possible room for improvement, and so on.
Eventually the duty passed to another set of hands, a bunch of students who had never seen Perl (the rise of PHP was strong at that point) and in the follow-up, they mentioned that they found it very straightforward to simply re-write each line.
I heard back in the past few months about one of those very large projects I did in a previous job, just keeping touch with people. I asked about one of my personal pet projects. "No, it just keeps running." "It's very obvious where to make any changes." All Python. No type stuff.
I just don't see what I am missing.
I understand generics have helped here, but they still don't seem quite there. And ironically, I'm finding ML tasks in python to be something that could really benefit from type hints.
Side note, and this bugs me: if people love it - empirically, they do - and you don't understand why - surely that should bring about some introspection? It seems all-too-often to bring about the opposite reflex.
Badly indented python code doesn't run, which it shouldn't - my understanding and the machine's are different. Well-indented python code has a lot less visual noise than other languages. That alone should give pause for thought. Why would you not want that feature?
> Why would you not want that feature?
Easy. Braces/etc. become basically invisible after using the language for a bit so they don’t bother me whatsoever. However moving code around in Python is always a pain because you must make sure everything is placed at the correct indentation level rather than simply letting the formatter take care of it. I’ve definitely experienced bugs from moved code having its first/last line at the wrong level, and it can be particularly confusing when the last line is at a different indentation than the previous. It’s so much easier to just grab a brace-enclosed block and smack it down somewhere (which also provides a good sanity check that you’ve yanked the entire block and aren’t missing any lines).
> Badly indented python code doesn't run
I wish this were the case. What’s actually the case is that badly indented python code will give you potentially different results than what you expect, which ranges from syntax errors to failed tests to very hard to diagnose bugs.
> Side note, and this bugs me: if people love it - empirically, they do - and you don't understand why - surely that should bring about some introspection?
Side note, and this bugs me: if people hate it - empirically, they do - and you don't understand why - surely that should bring about some introspection?
I know I set myself up for the last line. I can only say I've really made a good faith effort to try to understand the explanations for braces, and none make any sense to me. I've had nearly ten years writing more or less python alongside braced languages (I agree it's not a massive pain), and outside of the REPL - I have literally never seen an IndentationError (err.. that's the same as "doesn't run" to me), or "hard to diagnose bugs". Almost never a TypeError, either. Maybe a dozen or so?
I need to understand, from source code, how the instructions flow. For that, I and essentially all other humans need indentation. I never want the machine to interpret the instruction flow differently to me. I genuinely cannot understand how someone can fail to produce python code to that standard. I trivially can in any language which includes syntax specifically for giving a machine a separate understanding of program flow to humans.
Because no matter what problem you have, after 12 minutes of googling, you can pip install foo. The language almost doesn't matter because most programming is via library api.
Projects in the npm ecosystem may be somewhat more likely to.have typings if the project exists, but I find that, for anything other than web frontend, where JS is obviously king, the tool I'm looking for is more likely to actually exist in the python ecosystem.
Automatic stub generation may also be helpful [3]
[1] https://github.com/python/mypy/wiki/Creating-Stubs-For-Pytho...
1: among the most important things here, in addition to the strength of the type system, where mypy also lacks
I don't think that quite explains it. We loved Python at version 1, before pip and PyPI were around - although the "batteries included" standard library played a similar role to some extent. For me it's mostly the syntax and the simplicity.
You get used to it somehow. I did C and Java for about 10 years before Python. My first exposure to Python was a Computational Physics class and it was maddening not knowing what types went where. Eventually you realize it doesn't really matter as long as you're disciplined and rely on others who are disciplined enough to document their code well. I haven't had the chance to use mypy in the 7 years I've been using Python (across 3 industries; medical devices, scientific devices, and web) but I would be surprised if it significantly reduced the number of bugs I've seen in practice.
This is especially big for any refactoring work - just yesterday I had to change the format of a config file that gets read by both TS and Python scripts. For ts, I updated the typedef and I instantly got every error in the codebase annotated. For python, I had to manually go through each line, using my own brain as the type checker. My brain is much faster and better at being a “squiggly red line spotter” than a type checker, so I finished my TS work in a few minutes and took probably an hour going through the python, despite the projects being similar in size.
My experience with Python goes like this:
"I have to write a really small script... ok, not so small that I can do it with bash. I know, I'll use Python! This is going to be a small, throwaway utility anyway"
(Some weeks later, it's turned into a large monstrosity and I need to refactor something, and everything breaks because refactoring anything nontrivial written in Python is a dangerous proposition)
"Ouch! Python: never again!"
(Repeat a few weeks later)
https://hackage.haskell.org/package/turtle-1.5.19/docs/Turtl...
I do agree (and have the scars to prove it) that this isn't a panacaea but it does help, a lot.
That being said, one of the worst bugs I had in python was in passing in the wrong types to a constructor function, and my tests didn't catch that. To be fair, Mypy would have, but I hadn't annotated that part. At the end of that debugging situation, I would have gladly killed somone for enforced static types in Python.
That being said, the data model is a work of art, and a core reason why I enjoy coding in Python. It's just a shame that pandas kinda sucks.
- Most Python programs I write start their life as tiny scripts, always with the certainty they'll never grow (Narrator: they always grow). I don't know many people who write tests for their scripts...
- Testing in Python means too much effort in the wrong places. Consider that most refactoring problems would have been caught for free by a language with static typing. I've experienced runtime errors because I changed the return type of a function from a single value to a list (or viceversa). A statically typed language would have let me know of my mistake for free, so that I could focus on tests that really matter.
How do the benefits you get, relate to the difficulty of problems you’re solving when programming?
Eg if the cost of types is increased coupling of code but the trade off benefit is it makes fixing typos easy then that’s a bad trade. You’ve made a hard problem more brittle in exchange for making a trivial problem even easier.
> trivial problem
Type checking is quite non-trivial, especially as the logic and types get more advanced (conditional types, index types, mapped types, etc etc). Not just “typos”.
float multiply(float a, float b) { return a * b; }
Before you reach for the polymorphism or further pollute the universe of types consider that it costs less lines of code to erase types here.
What errors would preserving types help with here? What has more utility, less lines of code spent for the same outcomes or an implementation that exists further along the spectrum of typing in the “at compile time” direction? - I don’t believe that question can be answered but in practice i find less lines of code correlates more strongly with outcomes i care about than the degree of typing applied.
Pretty simple, it helps when you do a refactoring that changes the type of the value passed somewhere from number to number[] and the compiler instantly tells you that that’s illegal.
I did something similar just the other day when I had a shared config file read by both TS scripts and Python scripts and I needed to change the format to support a new feature. On the ts side I updated the typedef and the compiler pointed out to me every area that needed updating, on the Python side I had to spend a good chunk of time manually going through the script to figure out what needed updating.
Typing isn't a strong enough tool to combat that class of problem; you can fix the type signatures while introducing a semantic break. You need tests and if you have tests, what does typing bring? Keep your tests fast, measure and maintain a speed of >250 tests per second and remember to test behaviour not implementation - you won't go far wrong.
IDEs have been performing refactoring changes on our behalf in dynamic languages for years at this point. Don't manually edit when we've never had such powerful tooling available to us
* https://www.jetbrains.com/help/pycharm/product-refactoring-tutorial.html
* https://www.jetbrains.com/help/pycharm/structural-search-and-replace.html
Typing has a time and a place, that is without a doubt but the world is better viewed without the static typing lens permanently affixed in place, your efficacy will increase.
I’d never choose to have myself do more work if it’s not needed and the computer can do it.
Those automated refactorings would Not have helped in the situation I mentioned above.
It’s whether it continues to work tomorrow after you’ve added or removed a behaviour. Tests are the secret sauce that keep the cost of change low.
Refactoring (a commit with zero changes to test code) is the other key pillar of long term codebase health.