Lots of companies successfully manage far larger Python codebases.
We have a 35M LOC Python codebase with 2500 active developers making 25000 commits per week and continuous deployment globally of new code.
(I gave a talk on it at PyData London in April: "Python at Massive Scale")
The reasoning, a gigantic code base means infrastructure and tooling investment will obviously pay off. It probably even means you get one or more dedicated resources of this sort. Whereas a small to medium sized codebase might bump into some of the same issues as a larger code base but not justify such investments.
Do you have any extra writeup to suggest?
A good next step would be reading the 2016 ACM paper "Why Google Stores Billions of Lines of Code in a Single Repository" [1] then reading a recent HN discussion [2]. That has a ton of good links.
[1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
- Yes, it is a monolithic application architecture. There are distinct frameworks (e.g. GUIs, DBs, distributed compute, pubsub, lazily evaluated graph, reactive graphs, etc) designed to be used together in a unified way based on a global object store.
- A 150 person platform dev team writes these core frameworks and dev tools and runs the SDLC infra.
- Then the several thousand developers in business teams build applications for their particular needs.
- When the platform team sees businesses building their own solutions to a common need, it will productionize that as a new core framework for all business to use.
- In Python terms, there is a relative small number of core classes, and very many application-specific subclasses.
What I like about Python is the ease of which you can layout a project to fit your needs. Maintaining a large codebase is more about layout/architecture than the language itself.
Maybe other languages are simply better suited for programmers who lack the need of keeping their codebase clean and tidy :)
:)
- Where is this method being used? - If I delete or change this method will anything break? ....
The main issues with large codebases are often not feature-based but rather maintenance and operational overhead related: undiscoverability, config hell, dependency hell, and scaling issues - and typing helps a little bit with one of those.
If you delete a method that has usages without searching for them, you're asking for trouble. Sure it's not a compilation error in python, but compilation can only show that the method is missing, not that changes are guaranteed to work.
If I delete a method, I automatically get a big red dot in the corner showing compilation errors (without compiling).
If I change a type from string to int in both my method and method signature, it automatically tells me every place that would break.
How do you find usages in a provable correct way in a dynamic language?
I can’t understand why developers would want to take a tool out of their tool belt like static checking.
I’m a fan of Python as a low overhead scripting language but for large systems with multiple developers, static type checking is a godsend.
How much of it is generated?
To be fair, Django is pretty good: iirc, it induces good discipline for development, which is perhaps why it is so easy to handle a large codebase.
On the other hand, the business logic is outside the scope of Django (or any other framework), and there, if the devs do not follow good practices and develop the right tools, it might be more difficult to be productive.
The way I see it, people using dynamic languages have much more freedom in development, but also less safeguards, which means that they have to build them to make their code more robust. Having code generators is a way of getting back some safety, because you only have to make sure the generators are producing correct code.
Custom code, however, need to be thoroughly tested to be considered safe, and if the compiler doesn't do any type level consistency check, it's that many more tests that needs to be written, which adds up to the LOC count.
Code generation has so many downsides and Python is so dynamic I can't think what you're doing that couldn't be done better without code generation.
The rest of my life will do fine.
I'm sure you're right, but just for fun I'll make some counter-arguments.
All technologies follow an S-curve in their 'bang for buck' life cycle. Initial improvement and return on investment ramps up slowly, then it hits a middle era where all the low hanging fruit get picked for massive returns, then it enters a third stage where most of the easy benefits have been reaped and further improvement is incremental.
We see this with the internal combustion engine, jet engines, nuclear power, all sorts of thing. Eventually another technology might come along that supplants the old technology for many or most use cases.
The question I'm pondering is, are languages technologies in this respect, or are programming languages in general a technology? If the former then Python will top out in the benefits it provides and will almost certainly be supplanted by a successor. If the latter, maybe Python is near the top of the development curve (along with many other modern languages) and further improvements will be too incremental to lead to anything supplanting it?
But if that is the case, what technology might supplant programming languages?
One possibility is visual languages that emphasize mistake proofing, interchangeable parts, and logic visualization over expressiveness. I’ve noticed an increasing number and use of visual languages (Blockly, Unreal Blueprints, NoFlo, WebGME). These are typically criticized as too limited for real programming, but seem to be climbing the S-curve.
A more likely near-term possibility is continued adoption of systems that emphasize package management, which values reuse of proven code over ability to solve problems from scratch. It seems like the language package managers are a big draw in Python & NodeJS. Maybe this trend moves even farther toward stuff like IFTT, where most people working with code are really chaining together applets.
We’re also seemingly headed toward the Star Trek TNG computer where humans query a computer with voice commands and work with it to solve problems.
My point: Likely, there is a project size where Python's dynamic nature becomes a challenge. But 10 kLoC isn't it.
[1]: https://fman.io
I'm not arguing that it's not possible to do it; I just think that it requires a bit of extra steps and care to document which assumptions your code takes and how it behaves that could be more meaningfully described by a statically typed language. But these are just my 2 cents.
[1] https://github.com/RussBaz/enforce [2] https://news.ycombinator.com/item?id=18107818
Since Python 3.5 you can use type annotations:
def my_func(foo: str, bar: User) -> None:
baz: int = User.qux
In the long run, I'm sure some org will come up with a decent Python compiler, that:
* compiles Python to C
* makes use of (optional) type annotations
* improves performance
* creates single file binaries
I guess this would be less of an effort than porting the whole Python ecosystem to e.g. Julia.
The ease and safety you can make huge refactors is a huge win. Sure you can do it in python, but you will be writing x10 the amount of tests and final LOC counts including tests will probably have python come in closer to Java than the 10k > 100k loc indicates
I'm not sure what you're getting at but code management in Python is really no harder than anything else. It can be clean and well delineated, or you can completely screw it up... But that's not Python's fault.
I love Python; that's the first language I've learnt and I can't really live without its REPL. I only think that the industry has a tendency to excessively obsess towards certain languages, even when they are not the best tool for the job. For instance, I strongly believe that the academia's morbid obsession with Java has crippled a whole generation of new programmers, which are now unable to think and code in no other way but the Java™ Way. I won't say that python is as bad as Java in this regard (it isn't even close), but it annoys me to death when it looks like we're heading once again towards a language monoculture; at least this time the language isn't utter crap.
Isn't that's only true for data science?
What about the JavaScript monoculture :(
And academia still has courses for C java C++ etc. No respectable University teaches a single language!
Not like it solves anything -- just that there's this side of the argument too..
> I've seen horrible stuff, such as C89 still being thought like it's the only standard available, or gets()...
Youre absolutely right -- although one could argue that the curious will have the decency to look this stuff up (like myself)
The teaching system is still pretty rekt on the python side of things. Its not like you'll find people who actually code teaching at Universities anyway :(
Like now we're just going into how bad the teaching system is. The original argument that python is reaching is a monoculture seems weird to me, when I can't even run the damn thing on a browser..
At the end any new technology trend only has only 2 - 3 years, before people flood it and dilute both quality and wages.
The education system is another mess, which no one can clean up because the education system is designed to address literacy skills at scale(quantity), not quality.
Other big problem in the Indian system is endless optimization towards exams/interviews. Go to any OJ site on the internet and you see people even from Ivy Leagues(IITs etc) and NIT's flooding the site all day doing interview related algo work instead of focussing on their day jobs.
Its not just Python, tomorrow some thing else could be famous and you can see thousands, may be even tens of thousands of people learning it in India in a few months/years. This is regardless of the merit of the technology.
People really do need to be taught how to make out buzzwords from real tech; and the major responsibility falls in the hands of universities.
Maybe it's just because most people are just trying make their way out of poverty, and this kind of work _looks_ like a sure shot, get out of slum ticket.
Maybe it's because most people are led to believe that money is the only thing that matters..
It is much easier to decouple something in a dynamic language. The re-factoring needs in a dynamic language is thus entirely different.
Also, the de-coupling and modularization of functionality has been a thing long before dynamic languages became so popular. You simply need an FFI, and then the compiler makes sure that everything is straight before you get anywhere near deployment.
I think that the real problem is not having a good test coverage, and I agree that bad programs written in Python may be harder to work with than bad programs written in a statically typed language (but in a good codebase, even if large, I'd rather be working with Python).
-- although note that in the latest incarnation of Python you can try to use Mypy (http://mypy-lang.org/) along with https://github.com/Instagram/MonkeyType or https://github.com/dropbox/pyannotate to add types to you and go from there if you really miss types too much and this could make a bad codebase bearable (I'm personally waiting for PEP 544 before adding any types though as I'm not a fan of the type system implemented in PEP 484 very much).
Refactoring in a non statically typed language cant be as straightforward - many refactors in statically typed language can be automated and guaranteed safe.
I've always found C++ much easier to work with, if not to actually write.
Refactoring a python project is usually one man's job.
Refactoring a Java project takes a whole department and process management crew.
- rename/move class (and autoupdate all references)
- change method signature
Sorry but you are wrong. Just plain wrong. There are benefits for both languages. But to say python is easier to refactor is like saying Java is more concise. It isn't.
>Refactoring a Java project takes a whole department and process management crew.
Has nothing to do with the language, but the company. A small company will refactor and have an update out same day in Java. An enterprise using some python will be as you say
The cool kids of the future ain't going to learn Java once companies move to a better technology. It may take decades, but eventually people will stop building new code in the language.
a) Java is a bad technology and it's not moving forward ("once companies move to a better technology")
b) new Java developers aren't being trained, including very young and bright ones, straight out of universities, and they don't enjoy Java
Both claims are false.
My previous job we had about 3 million lines of Python and it was fine.
There are efforts to have gradual typing suppprt in python though, and you might be interested in having at least partial typing suppprt.
Take dart for instance -- it doesn't require any typing as such, but has an excellent type system that supports generics too. The type checker is pretty robust, but flexible at the same time...
GNU Emacs is 1.5 million lines of LISP code, another dynamic language.
It is ultimately a question of sound software architecture. You don't have to care about how many lines of code is in the libraries you use daily or in the Operating System you use.
In similar fashion any software can be modularized so you only need to concern yourself with your part.
I, along with one other guy, maintain a >500k fortran 77 code base (bit of C/C++ sprinkled in) at work. It's way cleaner than my home-brew python. Maintainable is about wise practices, not language choice.
I can't speak for Python but since the codebases were Perl and Common Lisp based, I believe it would be similar, should there be anything similar to the MOP for Python.
Static typing is great but not essential when the program is small and you can keep the model in your head, but the larger the program, the more you benefit from static typing. Static typing helps with tiny refactorings, but it truly shines when making non-trivial changes that don't play well with the existing model; it's as effortless as making the desired changes to the model, then mindlessly fixing all compile errors, and it will likely just work afterwards.
That's the reason why they attempted writing a new runtime first (Pyston), and then settled down on rewriting their performance critical infrastructure in Go (https://blogs.dropbox.com/tech/2014/07/open-sourcing-our-go-...)
Google iself wrote grumpy for the same reason, which is that CPython scales pretty bad.
I would love for a real contender to arrive, that challenges python's level of abstraction in a real way...
Nowadays, I don't think any new, dynamically-typed language has a chance to become mainstream.
They still look kinda low level to me. Yes, they provide a lot of high-level constructs and data types like python -- But what they fail to do (IMO) is a consistent abstraction.
For example, all of these languages have a concept of pointers. All of them have a distinction between different integers (int32, int64 etc..)
I had similar feelings when I used dart or even java. They're high level, but not quite!
I don't mean to thrash on these languages in any way. I think they have some real value for programmers.
It's just that I've never seen a language just throw everything out of your way and provide you with this extremely high level and easy to reason interface.
Nim actually looked pretty good, but then I realized it was just the significant whitesapce xD
Maybe I'm too biased because Python was my first (proper) language.
https://www.destroyallsoftware.com/talks/the-birth-and-death...
It's VERY real
Would love to try out something new.
typically languages like Haskell, OCaml are based on great abstractions (and are good to express new ones, especially from a type-system point of view).
If you're looking for languages which are similarly "easy to use", I can suggest to try Nim or Crystal.
Nim looks a lot like Python(indentation-based and similar constructs), it has a nice type system and very strong metaprogramming which helps to often express stuff in <= lines you would in Python (and also C and JS backends with excellent interop.
Crystal is inspired by Ruby, but it's also statically typed and mostly used for web development from my impression. Its windows support is still under development but it's also very expressive.
The point that I'm trying to say is that, while we're living in a moment where python is at the center of everything, it won't last forever; it never does. Something better always shows up, given enough time.