News for hackers. Stuff that matters.
Re:Our journey to type checking 4M lines of Python by jdm2212 on Saturday September 07, 2019 @07:30PM (#20907399)
The worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble than all the Java and C++ and Go combined.

My personal experience with compile time type checking is that the benefits kick in for a single developer project within a week, and for a multi-developer project immediately.

Re:Our journey to type checking 4M lines of Python by dbcurtis on Saturday September 07, 2019 @10:18PM (#20908181)
I am not going to dispute the value of compile-time type checking. And I admit to only having had time to skim the link at this point but it looks like a great read so I bookmarked it for later. That said...

Python can be as strongly typed as you like it to be. But of course, with Python, instead of insisting on an incoming parameter being of a particular type, the thing to do is Pythonically ask nicely if it can be the type that you want. If you need parameter x to be a float, call float(x), which will raise TypeError or give you a float. And of course, it gives the implementor of the class the option to implement __float__().

With a small amount of effort, you can design all of your __init__() constructors to do that style of validation. Yes, it requires some elbow grease. But you will not be able to pass bad stuff to my __init__ constructors if I don't want you to. And if you want some parameter y to be an instance of Foo, just call Foo(y) and if it is already a Foo, you get the Foo, and if it can be made a Foo, you get a Foo, and if it can't, you get TypeError.

So of course you actually need to exercise the code also, which is where the hypothesis package comes in.

Anyway, I think one of the reasons Python is popular is that it is very consistent about types. Haskel and Rust are also very consistent about types -- most other languages, not so much. Give me one end of the spectrum or the other. The murky middle is suboptimal.

Re:Our journey to type checking 4M lines of Python by joshuamorton on Saturday September 07, 2019 @11:18PM (#20908396)
This doesn't work as well as you'd hope.

It requires among other things, a unique and toilsome coding style, constructor overloading (which isn't a first class citizen of python), and you only get late runtime validation (which means, for example, you can easily miss type validation on only one side of a conditional).

Something like pytype detects such errors without any code modifications.

Re:Our journey to type checking 4M lines of Python by Too on Sunday September 08, 2019 @01:48AM (#20908819)
You can get compile time validation for this type of construct using typing.Union.

But once you have type validation i don't really see the use case anymore. Common example of how silly this is is functions that have a special case for handling lists of length 1 by allowing the object to be passed in without being part of a list, eg Union[List[Foo], Foo]. Now you need extra code to handle that special case in your function and the type annotation becomes more verbose. Much easier to just enforce List[Foo] as the only way to call your function and let the caller call you function as do_stuff([x]) if they only have a single item x. The IDE will complain instantly if you try to call it as do_stuff(x) so there is no risk to accidentally call it like that.

Re:Our journey to type checking 4M lines of Python by dbcurtis on Saturday September 07, 2019 @11:48PM (#20908497)
Most of what you say I agree with. But it is actually very easy to construct a __init__ that handles a very flexible parameter list. That style is well represented in the standard library. And I find that writing a beefy constructor simplifies parameter validation in methods.

But certainly, it doesn’t remove the need for good test coverage. Now, if you want to argue for less testing, that is a different pitch.

Re:Our journey to type checking 4M lines of Python by joshuamorton on Sunday September 08, 2019 @12:23AM (#20908586)
Possible, yes. Easy and straightforward, no. Casework and duck-type checks over args and *kwargs are not easy to maintain going forward, especially for a non-expert in Python.

Factory methods are usually better, and I believe what the more modern parts of the standard library have tried to use, but they don't work as well for this specific case.

Re:Our journey to type checking 4M lines of Python by musingsole on Sunday September 08, 2019 @01:26PM (#20911904)
> Casework and duck-type checks over args and *kwargs

Once you start doing this in python -- beyond any low complexity usage -- you're better off capturing your arguments in their own object and passing this to the function as a single argument. This object can then enforce whatever you want, and your function can now be a class method.

And then you can build a factory for this object that accepts a few args, maybe some keyword parameters. But, once that gets reasonably complex, just rinse and repeat. It's objects, methods and factories all the way down. And unlimited billable hours.

Re:Our journey to type checking 4M lines of Python by pilif on Sunday September 08, 2019 @04:04AM (#20909202)
> which will raise TypeError

At runtime. In production. A compiler would catch the problem at build time, never causing an error in production.

Re:Our journey to type checking 4M lines of Python by dbcurtis on Sunday September 08, 2019 @02:04PM (#20912182)
Maybe you missed this part:

> So of course you actually need to exercise the code also, which is where the hypothesis package comes in.

These are tests you should be doing anyway. Type checking does not excuse poor test coverage.

Re:Our journey to type checking 4M lines of Python by kalleboo on Sunday September 08, 2019 @09:17AM (#20910398)
> A compiler would catch the problem at build time

And a nice IDE/code editor would catch the problem as you're typing it

Re:Our journey to type checking 4M lines of Python by pilif on Sunday September 08, 2019 @09:23AM (#20910434)
Not in a dynamic language. Finding all possible inputs to your method that’s calling float() on its Arguments in a dynamic language is akin to solving the halting problem.

Also, error messages in an IDE can be ignored. Compiler errors can’t.

Re:Our journey to type checking 4M lines of Python by segmondy on Sunday September 08, 2019 @12:44PM (#20911607)
Why didn't you just rewrite it? If I have 500 lines that is a mess. It's getting rewritten. If you don't understand it enough to rewrite it, then you probably don't understand it enough to even maintain or change it. In my opinion, first thing I will do is dissect it till I understand it, rewrite it and move on. 500 lines of code ain't shit. I will kill the monster before it grows.
Re:Our journey to type checking 4M lines of Python by siempreb on Sunday September 08, 2019 @08:34AM (#20910166)
> The worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble than all the Java and C++ and Go combined.

That cannot be because of missing types. If you make just 500 lines python unmaintainable you're just a terrible programmer! In it's core it cannot be a type problem. And that is my main point here; there is a hype going on in the dynamic languages to add types, as if it solves everything. IMHO it's not that simple.

To avoid unnecessary explanation about the value of static typing; I am not against, I'm doing c++ and it serves me well.

Re:Our journey to type checking 4M lines of Python by jdm2212 on Sunday September 08, 2019 @11:37AM (#20911151)
The code in question was somewhat experimental, didn't have a clear owner and was modified by a number of people fairly frequently. There would've been much less room for sloppiness and shortcuts (let me just stick this extra field on this object) in Java, C++ or Go.
Re:Our journey to type checking 4M lines of Python by lawn on Sunday September 08, 2019 @12:45AM (#20908670)
At our job we use a homebrew C++-like language. Now some people are planning to rewrite a core part of the problem to be completely dynamic, only using key-value to control the behavior of a large number of things. There is no namespaces and the value is an Object (base class for all classes), so you always have to cast.

The absolute horror it will be to debug this mess when people starts to use it, my god.

Re:Our journey to type checking 4M lines of Python by realusername on Saturday September 07, 2019 @11:07PM (#20908363)
> The worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble than all the Java and C++ and Go combined.

Another problem with Python is that the culture of testing is definitely not as strong as in Ruby so you won't even get that to compensate the untyped nature of the language.

Re:Our journey to type checking 4M lines of Python by btown on Sunday September 08, 2019 @01:14AM (#20908742)
And yet another problem is that while Ruby classes require you to whitelist attributes (e.g. with attr_accessor) that will receive publicly-accessible getters/setters, Python classes by default allow the caller to "tack on" any instance variables they desire. (I've rarely seen __slots__ used in the wild, for instance.)

So the type of almost any class is really more like Struct{WhatYouThinkTheClassIs; ArbitraryDict;} ... and the temptation to misuse this, say for instance if you don't want to add arguments to function_used_lots_of_places(some_object), can be huge. And more generally, this openness means that there may be discrepancies between what's unit tested for a class, and how that class is used in the wild.

Re:Our journey to type checking 4M lines of Python by wutbrodo on Sunday September 08, 2019 @12:28PM (#20911505)
I'm still astonished that large companies run large systems on Python. I don't doubt that they're making the right decision, but it's a testament to how powerful inertia in large engineering systems is,given how incredibly horrible Python is for writing maintainable, sustainable multi-developer code. The most unfortunate part is that every engineer on a Python system has to be much more talented and disciplined to keep a workable system than they have to for statically typed languages (esp memory-safe ones), and yet one of the main attractions of choosing Python is chosen is that any rando can be nominally productive in it, which is helpful for talent-starved small companies.
Re:Our journey to type checking 4M lines of Python by soulnothing on Sunday September 08, 2019 @12:40PM (#20911587)
The early portion of my career was python. Coming from OCaml/Haskell. I couldn't really deal with J2EE.

But I've seen a frustrating trend recently. Where companies on languages like C#/Java. With a massive code base are moving to Python. I often get recruiter calls to lead a new python team, transitioning away from (Java/C#/etc).

The sole reason, is it takes to long to develop a feature in the legacy language. Which is resultant of outdated software and practices. I generally try and steer them towards CI/CD and newer lighter frame works. But someone way up heard python was the hot new thing. Now all critical back-end services must be in python.

To add to this I pushed for type hinting. But type hinting will "slow down developers". Part of the problem to me, at least. Is the person dictating development frameworks, languages, etc. Isn't really a coder, and heard about thing X at some conference.

I like python, but it's an infrastructure quick scripting language. Additionally a lot of the good python devs I know shifted over to tensor flow and machine learning.

Re:Our journey to type checking 4M lines of Python by ken on Sunday September 08, 2019 @01:02PM (#20911714)
Counterpoint: I've worked on a production >100-KLOC >10-programmer Python system, and lack of types was not an issue at all.

For the past week I've been updating a (much shorter) solo app, written in a statically-typed language, and having a bear of a time getting the compiler to stop throwing false type errors at me. Yes, it's the stable release version of the compiler. That has never stopped any compiler from shipping with bugs. These type inference systems can be pretty complex.

Then again, I seem to run into more compiler bugs in general than anyone I know. How do you all do it?

Re:Our journey to type checking 4M lines of Python by tus88 on Saturday September 07, 2019 @07:41PM (#20907459)
How are you sure typing would have made a difference to that piece of Python? Are you sure there is no troublesome code written anywhere in Java or C++?
Re:Our journey to type checking 4M lines of Python by swiley on Saturday September 07, 2019 @07:46PM (#20907480)
IMO type checking is to python what bounds checking is to C.
Re:Our journey to type checking 4M lines of Python by speedplane on Sunday September 08, 2019 @06:28AM (#20909642)
> IMO type checking is to python what bounds checking is to C.

Interesting analogy, but in my experience it doesn't hold up.

If you have a Python typing bug, it will crash your program almost immediately and you can quickly catch it and fix it. In contrast, C boundary checking issues often come up in relatively rare corner cases, and when they do occur, they often do not result in an exception / error especially if the memory is already paged and set to zero.

In production code, boundary checking errors in C occur all the time, but Python type bugs are pretty rare.

Re:Our journey to type checking 4M lines of Python by justwalt on Saturday September 07, 2019 @07:49PM (#20907498)
I think he's talking specifically about the Java and C++ code that accompanied that Python file. I'm sure he's seen bad Java and C++ :)
Re:Our journey to type checking 4M lines of Python by jdm2212 on Saturday September 07, 2019 @08:01PM (#20907556)
I've seen some disgusting Java and C++, but that company had pretty high engineering standards so the Java and C++ was always good. The Python was _supposed_ to be good, too, but 20+ devs making changes to this one config generator script meant people were constantly making their own (and breaking each other's) assumptions about how the code should behave.
Re:Our journey to type checking 4M lines of Python by proverbialbunny on Sunday September 08, 2019 @12:16AM (#20908574)
At that point the first person to write good unit tests gets to specify how that part of the program is going to run. Though, really it sounds like you guys need a design pattern to address everyone's needs that everyone can agree to.

People can change unit tests. The benefit of this, is it's a kind of documentation. People can see the tests and know how it _should_ work, or they can not read them and if they create errors, they'll find out the hard way. Unit tests don't make a specification static, but they do reduce chaos.

Re:Our journey to type checking 4M lines of Python by jdm2212 on Sunday September 08, 2019 @02:12AM (#20908898)
This was a few years ago. The chaos was partly because of a lack of ownership: one team owned inputs, the other owned outputs, and there wasn't a clear spec. The whole pipeline was pretty experimental and people made up new semantics as needed, and were happy to change failing tests (or just added untested codepaths behind boolean flags in existing functions). Compile-time type checking would've enforced a greater degree of rigor about modeling what was happening and made clear all the ways existing code could be affected.