Code that assumed it was arbitrary, would expect to handle any arbitrary order, including a happens-to-be sorted order.
Code that assumed it was random, like actually inserted by random(), was already broken, because that simply isn't the case.
Code that assumed the order would stay constant was relying on implementation-specific behavior, and could potentially break on any version update; as with any reliance on implementation-specific behavior, you'd break if the dictionary code ever got touched -- even if it were for a bugfix.
Code that ordered the dictionary keys before iterating are now slightly innefficient due to extra work of sorting a sorted list.
I could potentially assert a version, but do I really want to do that each time I wrote something that might be used somewhere else?
And yes of course I could add a line of documentation, but there is a 100% chance I’d still get bug reports from people on 3.5.
If you are just distributing raw python files then congratulations you’ve just realised why packaging is valuable.
If you want to install it, go ahead and copy or symlink it in your ~/bin or whatever you fancy (that's your personal preference anyway unless I'd specifically package it for some OS like Debian). I don't want to have to use some setup.py that I have no clue where in my OS it installs things.
It's not duplicated and in most cases it's not even delivered as part of the installation.
> If you want to install it, go ahead and copy or symlink it in your ~/bin or whatever you fancy
That's exactly what pip will do if invoked with `--user`.
> I don't want to have to use some setup.py that I have no clue where in my OS it installs things.
It installs it to a single place. Run `python3 -m site` and look at `USER_BASE`.
To avoid a lot of this, use pipx[1] to keep things even more isolated.
> Often have to do chenanigans like "python3 -c 'from sometool import __app__'".
You're doing things wrong because you don't know the tooling. You'd also typically just do `python3 -m sometool`.
Things that are distributed as a single file are either so simplistic and have no other dependencies that you can just make do, or written by someone who doesn't know what they are doing and so you're going to have a bad time.
If your code won’t work for older versions you can make an explicit check that the version of python is greater than whatever you need.
I suppose Python doesn't even have backwards compatibility within the same major release as we saw with the addition of the async keyword in Python3.5. Many older Python 3 packages broke because they expected that to be a legal identifier for a variable name.
facebook is just as capable of writing hot garbage, sadly.
But in the case of downgrading, I'm fairly sure there's a number of other breaking changes that can't trivially downgrade minor versions. Like f-strings were only introduced in python3.6 as I recall. Async keyword only exists as of 3.4 as well I think?
The only way to consistently code cross-version is to start with the lowest you plan to support (assuming the higher versions are actually backwards-compatible).
Does any language gaurantee that code is both backwards and forwards compatible?
you arguably ought to anyway, for explicitness.
It has useful features for manipulating ordering but while I've regularly needed had use for maintaining insertion ordering I can't remember ever needing to move items around within a map.
It remains so today, ordereddict is not an alias or trivial facade to the normal dict because it has to maintain its doubly linked list and implement a bunch of additional operations based on that e.g. pop first or move to end.
>>> isinstance(OrderedDict(), dict)
TrueThat said, Go made a similar change (from insertion-order to explicitly-randomized) and world didn’t end. So there’s that.
That field itself is kinda new; but if needing to block users with older versions, that shouldn't be an issue.
See https://medium.com/i0exception/map-iteration-in-go-275abb76f...
That said, I have no idea about the internals of dict. I assume no performance was sacrificed for this change.
Also, this change was implemented 3.6, but in 3.7 they officially documented it as a language feature (i.e. that all other Python implementations also need to preserve the order).
That's a truism. For all versions of Python. If you use feature of python ver X, you should not be surprised that it doesn't run on versions less than X that lack that feature!!!
If you write a Python library and use feature of Python ver X and don't mark library as only >= Python ver X, you are doing it wrong and a horrible person.
I don't see how anything could break (unless there are alternative implementations with different behaviour I'm not aware of?)
Dicts have been effectively ordered since 3.6. Iteration order was literally randomised (at process start) before 3.6. I'm also not sure whether the behaviour under deletion was changed between 3.6 and 3.7 so it's possible that there are subtle differences there.
The other why around is also true, code that relies on it must specify that it requires python >= 3.7
If they claim compatibility with only up to python3.6, they can have whatever order they choose.
The only issue with portability is that I think the main reason it was made a gaurantee is that cpython found the new, presumably optimized, implementation came with insertion order for free, so they went ahead and gauranteed it. But that might not be an optimal strategy in other areas, but they're forced to follow along anyways.
But actually moving cpython code to say ironpython should not be impacted, unless ironpython lies about it's compatibility
I get the whole principal of least surprise, but not at the expense of progress.
Dicts are ordered, specifically insertion-ordered, not sorted.
Sometimes they'll be sorted:
D = {i:i for i in range(10)}
D[-1] = -1
zip(D.keys(), D.values())
D.items()
> If items(), keys(), values(), iteritems(), iterkeys(), and itervalues() are called with no intervening modifications to the dictionary, the lists will directly correspond. This allows the creation of (value, key) pairs using zip(): pairs = zip(d.values(), d.keys()).
oldkeys = list(D.keys())
D[z] = foo(z)
zip(oldkeys, D.values())
The subtlety of this is what I, and perhaps others, find the most jarring.
Baking this as a language guarantee is the only protection against Hyrum's Law.
> Changed in version 3.7: LIFO order is now guaranteed. In prior versions, `popitem()` would return an arbitrary key/value pair. [1]
If they added a new `popordereditem()` method, good 3.7 code would use that, and an attempt to run that code on 3.5 would throw a reasonable, useful error message. If they wanted to play it safe like Rust or Go, they'd add the ordered method and make popitem() deprecated or make it artificially use random/arbitrary order so you can't accidentally depend on a new implementation detail and have a test case work.
Also, your case 4 does deserve some protection because while bad code it's hard to test for bad but working code. Implementation-specific or undefined behavior that works is the worst kind of problem, because it's hardest to test against. Compile-time syntax errors are the easiest, runtime errors are testable with a solid test harness, some possible can be identified as warnings with linters, but sloppy code requires manual inspection to detect.
Actually, no, I take that back: Implementation-specific or undefined behavior that works sometimes! is the worst kind of problem. That's what this code enables; if your test case is `dict({'one': True, 'two': True})` and you have a unit test where you popitem() to assert that you get 'two' and then 'one' it will pass the test harness on 3.7, it will pass on 3.6 because of implementation-specific behavior, and it will pass on 3.5 because the hash map arbitrarily puts those particular items in order. But it will silently get it wrong when you pass in user-supplied data. Shudder.
[1]: https://docs.python.org/3/library/stdtypes.html#dict.popitem
I think TimSort is extremely efficient for presorted lists, so even that isn't a major impediment.
But, from a pragmatic angle, it strikes me as a genius move. As the article said, this was a natural by-product of a performance enhancement that was made in 3.6. In principle, that was fine, because there was officially no predictable ordering, so a change to how it was being ordered in practice shouldn't materially affect anyone. And nobody really paid much notice to it then.
And that's where the danger lies - there's risk there, because, as you said, someone who's used to Python 3.6 might have problems switching to 3.5. And that's probably a greater risk if they left it out of the spec, because people would have continued to not pay it much notice. By making it official in 3.7, though, they've sent out a message that the Internet hate machine has ensured that everyone will hear. That probably, in practice, actually reduces the risk.
And, while many understandably see this as violating the spirit of semver, it unambiguously does not break the letter: Minor changes are only supposed to avoid breaking the software's compatibility with code written against an old version. There was never any rule saying that old versions have to be compatible with code written against the new version.
https://play.golang.org/p/DISpyv0Zuq_j
HN discussed this a little 6 years ago: https://news.ycombinator.com/item?id=7655948
As crawshaw pointed in that discussion, it helps catch people inadvertently relying on map order in tests or other places in their code.
That is actually why the Python maintainers decided to make this behaviour official: the ordering was the consequence of changes in implementation details, but over 3.6's lifecycle they feared it would cause compatibility issues as users would start relying on the ordering properties of CPython and that would be an issue for alternate implementations (though pypy had switched to the same implementation even earlier so was also insertion-ordered even when running in Python 2.7 mode).
Providing stronger guarantees was considered useful and unlikely to be severely detrimental in the long run, so the project decided to make it part of the language spec.
It does come at a cost, but I think Python aims for ease of use over runtime speed and memory efficiency, so it seems perfect for them.
This one is unusual because it won't break old code being brought forward, only the other way around, and theoretically there are lots of things going the other direction that would break (though most of them are explicit, not silent).
Backward compatibility is the ability to run old code with new interpreters. This is not broken here.
What's broken is the ability to run new code on old interpreters. But this is already broken at every python update (new operators, methods, syntactic sugar..). We could call it reverse backward compatibility.
- backwards compatible code: new code can run on an old interpreter
- backwards compatible interpreter: old code can run on a new interpreter
EDIT: After some thought, you're right. The second description is the reasonable interpretation.
"python 3.7 is backwards compatible with python 3.6"
(Driving to an airport) "Okay, we gotta pick a road. Arrivals or departures? We're arriving, but then we're departing."
This is like many evaluation guarantees in latest C++-NN standards. Good to know, might be useful for performance tuning of automatic code generators, etc. , but irrelevant most of the time. My 2c.
People will just do what they've always done - they'll be aware of the differences, or they'll test.
There's nowhere I have seen that mentions dicts are ordered without mentioning in the same paragraph the version since which this has been the case, so anyone who knows they're ordered will be aware.
Tldr its fine
If a new guarantee or behaviour were a breaking change, every non-patch release of a semversioned tool (which python isn't) would be major.
And randomised at interpreter start, since Python 3.3.
And that's sort of the problem. If it was broken, you'd know it and you'd fix/rearchitect it. Instead, it will appear to work.
For example, if you wrote something in pypy it would be ordered versus cpython.
I can't imagine how it can break something. In which case can you have an advantage to have an unordered list? Biggest downside can be about perf, but I don't think it's the case here.
It's important to discover coupling between your tests, this just isn't the way most people want to do it.
And beyond this, even if maintaining backward compatibility is important, I don't think it should prevent software from being improved, and dicts being ordered is pretty awesome. I'm pretty sure relying on unsorted dicts might involve pretty hacky code.
I recently stopped writing a z-order test application because I remembered that dicts are not sorted, so it made my goal a little pointless. Meanwhile C++'s std::map is sorted. I think there was already sorted dict in python, but I don't remember.
EDIT: I'm wrong, I'm mixing "ordered" and "sorted"
Dot-upgrades do allow for breaking changes, but this one can break silently and I’m not sure where such breakages neither belongs in python nor where they should belong.
Personally I’m happy to get ordered ducts sooner rather than later.
It's like, you have a great marriage with everything working out and some fine children who are doing great too, but you get a divorce anyway to pursue your dreams of being a conceptual artist.
(BTW, in case anyone is keeping track, I was going to maintain P2 but got caught up in Prolog, of all things, and now Python looks just as crude as everything else and I don't have the requisite love anymore to shoulder the work. I might make a Python 2 interpreter in Prolog though! That would be fun.)
Big whoop
Javascript doesn't specify performance characteristics of objects and arrays, so even with implementations, one object could be a hashtable, another a balanced tree.
In the past (from before this standardization) Chrome had in fact changed the object iteration order due to an optimization, and had to revert it after lots of complaining on their bug tracker.
(To be precise, the spec still requires array index properties to be returned ahead of other properties regardless of insertion order. The behavior that Chrome "reverted" to is this new one, so not exactly the same as the original behavior.)