What's the alternative? Deprecate warning old-style string handling and slowly nudge it out? Great, that's not actually decideable, since you need to know how a str is being used.
str are at the very heart of python. This would be a gargantuan task just to write the linter.
No, its often misremembered that way; Ruby 1.9 had lots of backward-compatibility breaking changes, there was a long time when lots of things were stuck on 1.8, and much of the other stuff that happened with the Py2 -> Py3 conversion was mirrored. The big difference was that the Ruby community and ecosystem were smaller and less diverse.
Huge chunks of industry and academia run python on windows (usually through (ana/mini)conda). That's a whole 'nother world of encoding hurt.
Especially around the 1.9 transition; Ruby was barely usable beyond the level of toy code on windows. The Ruby on Windows story is actually a lot better now.
https://thoughtbot.com/blog/the-journey-to-ruby-1-9
One key here is that the common subset of Ruby 1.8 and 1.9 was large enough that you could ship a single library for both, and a lot of large ones did, as in the above examples. That wasn't the case initially in the Python transition, and by the time Python3 had started making enough concessions to back compatibility to make this practically possible, people were a lot less willing to try.
(I personally was managing upgrades or a large Rails app at the time; Rails upgrades were, and remain, a bear, but as an accident of code style which just avoided the tricky spots anyway, the changes required to move that code between any two Ruby versions have never been more than a day or two.)
Lots of things; there was for several years a “caniuse” equivalent for Ruby gems tracking if they were on 1.9 yet because it was such a big issue. Bigger issue outside of the Rails-centered ecosystem.)
> I personally was managing upgrades or a large Rails app at the time; Rails upgrades were, and remain, a bear, but the changes required to move that code between any two Ruby versions have never been more than a day or two.
Yeah, a lot of Rails use exercises a fairly narrow slice of Ruby language and core/stdlib directly (that js, outside of library code), while leaning heavily on Rails-provided APIs, so that's not too surprising.
AFAIK Ruby's change was that they introduced a notion of encoding to strings. Python literally change the string types and its semantics.
Adding a bunch of other BC breakages was a good idea. Having been responsible for the migration of a large codebase the strings change was by far the most disruptive and problematic one, the rest did not even come close (the distant second was the change to rounding IIRC). Ripping the bandaid and adding a bunch of other BC breaks to clean things up or prepare for the future was I think an excellent idea. Most of those were easy to shim or work around (the syntactic BC breaks were especially trivial).
The problem, as GGP's comments correctly notes, was the incompatibility between Python 2 and Python 3: when Python 3 was first released it was impossible to write a non-trivial program which could run on both version.
Meaning if you wanted to port a project you had to convert everything at once. And as if the odds of that working were not infinitesimal enough, libraries had to fork their own codebases, and either try to maintain completely incompatible codebases concurrently or drop support for Python 2 while literally all their users were still on it.
Their point is not that Python3 should not have broken backwards compatibility, it's that Python 3 should not have been entirely incompatible with Python 2. That decision is what hampered migration for years, because it turned out not to be a feasible migration strategy, it would have required the entire ecosystem to migrate in lockstep which when you think about it made no sense.
Python 3 migration picked up steam once the core team backported what could be backported to Python 2(.7), and reintroduced syntactic compatibility features to Python 3.
At that point, and with a few shims for standard library stuff, libraries could create cross-version codebases, which meant they didn't have to try and fork their own codebases. And dependents could migrate at their leisure.
No, my point is precisely that py2 and 3 are not compatible. Even with the backports, py2 and 3 are not fully compatible. Hence the "py23" or "six" dialect, which is the subset of 2 and 3 that overlap, with some shims to facilitate it.
Py2 and Py3 string apis are not compatible. They cannot be made compatible. They can never be compatible. There's no "from __future__" that could make them compatible without a stupendous amount of work.
Not to be pretentious, but I'm developing the opinion that anyone who thinks that the Python team could have just shimmed their way between 2 and 3, does not really understand the Python compute model.
Google literally invented a py2 to go transpiler (grumpy) to avoid converting. Do you think for a second if there was a way to just "from future" their way to a compatible str object, they wouldn't have taken that route?
Which still misses the point, py2 and py3 not being compatible is different than them being completely incompatible, because there is a useful, workable, common subset of Python 2 and Python 3.
The entire issue with Python 3 was that there originally was not, the core team had to be dragged kicking and screaming into carving it out through backports and reintroductions because their original strategy was not workable.
> Py2 and Py3 string apis are not compatible. They cannot be made compatible. They can never be compatible. There's no "from __future__" that could make them compatible without a stupendous amount of work.
So?
> Not to be pretentious, but I'm developing the opinion that anyone who thinks that the Python team could have just shimmed their way between 2 and 3, does not really understand the Python compute model.
How can you claim that when "shimming their way between 2 and 3" is exactly what ended up happening?
> Google literally invented a py2 to go transpiler (grumpy) to avoid converting. Do you think for a second if there was a way to just "from future" their way to a compatible str object, they wouldn't have taken that route?
Have you considered stopping beating up your strawman?
Writing 2/3 python is harder than writing either 2 or 3. Porting 2 code to 2/3 is about as hard or slightly more work than just converting straight to 3. But at the end of it, the benefit is your code can run in both environments.
So let's say instead of pulling the plug, py leadership says "Python '2.8' will only run 2/3 code." How is that any different? You're gonna have the people who already ported, who have no need for 2.8 cause they can run 3, and the holdovers who have yet to port from 2.7.
That's what I mean by there's no shim.
> The entire issue with Python 3 was that there originally was not, the core team had to be dragged kicking and screaming into carving it out through backports and reintroductions because their original strategy was not workable.
Maybe I got into the game too late then, but I don't recall it being this much of a dust-up.
"six" was released on pypi as 0.9.0 on Jun 28, 2010. 1.0 was in Mar 2011. Even if the py core team was dragged kicking and screaming, that's still 4 + 5 years of people who could have been porting, and weren't.
Basically, my primary point is, it was a damn hard problem, and I doubt there's much the core team have done better at the time. All the porting tools in the world won't get people to upgrade if they are resistant for any reason.
E: py3 was released in Dec 2008. So I guess those first two years might have been tough. But to me that seems exactly like the ripe time to work on the translation infrastructure and things like six, where you have public release but no immediate pressure to switch.
3.2 was released early 2011, and P2 compatibility features were reintroduced as late as 3.5 (% on bytes) in 2015, though for my money the last key one was the reintroduction of the `u` prefix in Python 3.3, in 2012. That's the point at which the migration really became feasible without suffering.
Furthermore that's assuming all dependencies were ported, which was not the case: waiting until porting was feasible (and they had the time and manpower to) was also something they had to wait on e.g. Django first added Python 3 support in 2013 (IIRC PyLint only supported 3.4 onwards), difficult to port your own software if the stuff you depend on hasn't been ported over.
They should have made the unicode handling changes opt-in per package or file. Then, start warning about anything that hasn't opted in. Then, only after a significant migration period, change the default to the new unicode handling.
Nearly all of computing has made a similar transition in the 2000's and 2010's, and I've certainly debugged my share of encoding issues, but in general it worked for a large majority of use cases.
I think most people would've taken a few encoding issues over the 10+ years long migration.
Python 2 was a code pages world. Not an ASCII world. Also Python’s Unicode isn’t and hasn’t ever been UTF-8 anywhere; rather, it’s sequences of Unicode code points (note that that’s code points, not Unicode scalar values−this and the representation of str are a pair of things I find quite bafflingly bad about Python 3, because it means Python’s Unicode strings aren’t, y’know, Unicode strings, because they can be ill-formed, and they’re stuck with bad fixed-width representations so that any even vaguely interesting string will be stored as (ill-formed) UTF-32).
Python 2’s str type was a mess, used for bytes data, for strings of uncertain encoding, and for Unicode strings. In a vacuum, you might say you could make that work by splitting it up (and indeed the types module was an early endeavour to do that sort of thing back before it became evident it was insufficient). The key is that this confusion and widespread misuse was rife through the standard library and other third-party libraries. If the Python 2 → 3 transition had been just str-as-bytes/str-as-who-knows-what/unicode → bytes/something/str, it might have been manageable. But the libraries were so shot that a straightforward migration was impossible, because str was too many things and it was quite impossible to tell statically what should be done about it. (And impossible dynamically to resolve as well.)
Python would sometimes guess the output encoding on Windows to be ASCII even when UTF-8 was supported by the output device. So you'd end up with encoding-oblivious Python 2 programs that worked perfectly reading and writing UTF-8, and Python 3 programs that failed upon encountering a non-ASCII character.
They've since improved the defaults on Windows, but the changes in Python 3 that caused me a lot of pain were when they made IO to be non-Unicode.
They're still constantly finding bits of code that break cause some customer decided to switch from "delta" to “Δ” in some machine learning log output.
You can override the built-ins by re-defining the symbol in any of: local functions, enclosing functions, or the global scope and it will impact the entirety of that scope.
If I'm understanding your comment correctly this time, it would look something like:
getattr(thing, b'attribute_name')
getattr(thing, u'attribute_with_unicode_characters')
This would work in both python 2.7 and 3.9 (probably works with other 3.X versions but it's the one I actually tried).
str = int
Well python is a duck-typed language, so it will handle the unicode string as if it was a bytestring, and this may or may not work correctly. It's up to the modern code to interface correctly with the legacy code. This could mean manually converting between bytestring and unicode string.
Upgrading to a new major version should never require changes to your code if you fixed all deprecation warnings in the previous major version. Otherwise, you make incremental upgrades impossible.
The new alternative should always be introduced as opt-in with the old default deprecated, and then made default in the next major version.
Have an import switch that enables a legacy mode, that will convert your strings into and out of the library? You can even require that the application developers cast their strings into the correct ones if you don't want to calculate it on the compiler.
str is too deeply embedded and used in too many dynamic and unique ways for that approach to work without doing some really ugly or nonperformant or community-fracturing things.
And it might well be that the deprecation notice will run for two versions. At the moment no official decision has been made as it is still an active issue.
I disagree, I think you're learning exactly the wrong lesson from the Python3 debacle. It took a long time for Python 2->3 because the Python3 developers ignored the need for easy backwards compatibility, just like the Go developers are doing. Python3 required all software, including all your transitive dependencies, to be simultaneously updated. Python3 was dead-on-arrival for many years until there were finally concessions made in Python3 to make backwards compatibility less painful (in this case, so that code could more easily be written to work in both Python2 and Python3).
If the go developers want people to use "install" instead of "get", sure! But give time so that the change can be distributed across the ecosystem before removing the functionality. There's no hurry.