The primary one I want is
if m := re.match(...):
print(m.group(1))
and while s := network_service.read():
process(s)
both of which are both clearer and less error-prone than their non-walrus variants.The other one that I would have found useful an hour ago is in interactive exploration with comprehensions. I frequently take a look at some data with [i for i in data if i[something].something...], and being able to quickly give a name to something in my conditional, as in {i: x for i in data if x := i[something]}, helps maintain the focus on data and not syntax. Obviously it will get rewritten to have clearer variable names, at the least, when it gets committed as real code, and almost certainly rewritten to be a normal non-comprehension for loop.
Like comprehensions, I expect the walrus operator to be valuable when used occasionally, and annoying if overused. There's no real language-level solution to bad taste. In Python 3 you can now do [print(i) for i in...], and I occasionally do at the REPL, but you shouldn't do it in real code and that's not an argument against the language supporting comprehensions.
m = re.match(...)
if m is not None:
pass
Now, I wonder what you meant saying that single-line version is less error-prone, because I don't think so. I believe they're exactly the same in this regard, except for a bizarre case when someone would bastardize the code by putting some irrelevant lines between the assignment and the comparison, obfuscating the logic.Also, as I tend to use `if m is not None:` rather than just `if m:` (because I typically don't want to run all that subtle `__nonzero__` magic), it would - subjectively - look less pretty in the one-line version: `if (m := re.match(...)) is not None:`.
Use in the generator expressions to create aliases/shortcuts and avoid repetition is great, though. I love this use case.
while m := f(many, args):
# do stuff with m
Now, if you're writing this in Python 3.7, you often end up with some code duplication: m = f(many, args)
while m:
# do stuff with m
m = f(many, args) # duplicate
Or something like this: while True:
m = f(many, args)
if not m:
break
# do stuff with m
Personally, I consider the last version to be the most elegant, but still something of an anti-pattern. So I welcome this new operator.Edit: 1 line, not 2 (excluding the "do stuff")
def f_iter(many, args):
while True:
m = f(many, args)
if m:
yield m
else:
raise StopIteration
...
for m in f_iter(many, args):
# do stuff with m
This way you’re isolating all the initialization logic, error handling, etc. And you can focus on your domain logic in your client code. from functools import partial
with open('mydata.db', 'rb') as f:
for block in iter(partial(f.read, 64), b''):
process_block(block) def f_iter(many, args):
while True:
m = f(many, args)
if not m:
return
yield m> [bpo-32670]: Enforce [PEP 479] for all code. This means that manually raising a StopIteration exception from a generator is prohibited for all code, regardless of whether ‘from __future__ import generator_stop’ was used or not.
[changelog]: https://docs.python.org/3.7/whatsnew/changelog.html#id115
[bpo-32670]: https://bugs.python.org/issue32670
[PEP 479]: https://www.python.org/dev/peps/pep-0479/
for m in filter([re.match(...)], bool):
...
;-)Also if it comes to pure form I would have preferred `while (evaluate) as x:` out of establishing a parallel with existing syntax, but that's not very important.
In that sense, while loops over expressions always require knowledge of the expression to understand the semantic. On the other hand, something like:
while True:
with x as y:
print(y)
would not require understanding of x to understand the intent. Therefore it is not semantically equivalent to: while x as y:
print(y)
Note that in both cases, `:=` is strictly equivalent to `as` as it is defined for `with`. That is a matter of personal preference. for line in f:
if (m:= pat1.search(line)) is not None:
... do stuff ..
elif (m:= pat2.search(line)) is not None:
... do other stuff ..
elif (m:= pat3.search(line)) is not None:
... do something else ..
In older Python that's: for line in f:
m = pat1.search(line)
if m is not None:
... do stuff ..
else:
m = pat2.search(line)
if m is not None:
... do other stuff ..
else:
m = pat3.search(line)
if m is not None:
... do something else ..
I think the newer makes it clear that it's supposed to be a simple elif chain, where all branches following the same structure.There are other ways to structure it, but the alternatives I can think of also have their own cumbersome complexities.
LINE_ACTIONS = (
(re.compile("pattern 1"), do_stuff),
(re.compile("pattern 2"), do_other_stuff),
(re.compile("pattern 3"), do_something_else),
)
...
for pattern, action in LINE_ACTIONS:
m = pattern.search(line)
if m is not None:
action(m)
break
Or even use a method registry pattern that would auto-populate LINE_ACTIONS by just declaring the fuctions: @action("pattern 1")
def do_stuff(m):
...
Alternatively, I might just break the processing into a function: def _process(line):
m = pat1.search(line)
if m is not None:
... do stuff ...
return
m = pat2.search(line)
if m is not None:
... do stuff ...
return
m = pat3.search(line)
if m is not None:
... do stuff ...
return
_process(line)
Of course, this depends on the purpose. Could be completely inadequate in some situations.I've found that I don't like using it. As you write, it's "inadequate in some situations", which I consider as part of the cumbersome complexities I mentioned.
For examples, do_stuff() and do_other_stuff() may need to share variables, and do_something_else() might need the line number to report an error while the others don't.
This can be handled with shared state/nonlocal, and by passing in more parameters to the generic handler API, but these add complexity.
Or, different parts of the file may have different line dispatch processing (eg, a header block followed by a data block followed by a footer block) where one of the handlers must indicate the transition to a different processor.
Also, function dispatch in CPython is slow. While the regex tests are also slow, it can also be important to consider the (using a hypothetical number) 5% overhead for dispatching over inline code.
re.match() should always return a match object, but instead .group(1) will return None. Then we can write one-liners easier without the walrus operator.
The problem is not in re.match but in None being a relatively poor Maybe / Optional (still ways better than C-style `null` of many other languages).
if re.match(...).matched:
But yeah, agree that Optional is what I want here. if let Some(m) := re.match(...)
m.group(1)...
m = re.match(...)
m.group(1) # TypeError because I haven't unwrapped the Optional
In current Python, that last line sometimes throws a runtime error.re.match vs re.search
>>> import re
>>> pat = re.compile("(A+)|(B+)")
>>> m = pat.search("HUBBLE")
>>> m.group(1) is None
True
>>> m.group(2)
'BB'
>>> m.group(0)
'BB'
I don't see how using your proposed change would help.The inside of the for-braces allow for keyword-based sub-clauses: There's
- ":when" which will suppress output for elements not matching a filter expression,
- ":while" which will stop when elements stop matching a filter, and
- ":let" which will let you bind some new values in mid-loop.
I rarely need these features, but when I do I find them really really helpful. Maybe someday someone will consider doing something similar in Python.
for s in iter(network_service.read, ''):
process(s)... I've got no idea how the syntax could look though.
if (value := func()) and another_condition:
...
Or: if another_condition or (value := func()):
...
Or: x=1
my_list = [x := x+1, x := x-1]
etc.That was the reasoning given for using ":=" instead of "as", to allow more complexity. I still think it was a mistake.
Watch out with this one though. In this case it's good, but if you receive numbers from such function, it will break loop not just on None but on 0 as well. Easy to forget.
In many cases you'll need to do:
while (s:= ....) is not None:https://www.python.org/dev/peps/pep-0572/#differences-betwee...
https://www.python.org/dev/peps/pep-0572/#relative-precedenc...
Even examples of the spec shows how unintuitive and "unpythonic" this is. Explicit is better than implicit.
IMHO adding features to the language to save 1 line of code for 10% of cases when you need it (I agree that there's occasional case when walrus will save you more than 1 line) is just bloat.
I am not a big proponent of Go, because it has its own flaws, though language is indeed very simple and creators of the language try to leave it simple.
IMO Python was very readable, super simple, intuitive and should stay that way, though recent releases show that Python is giving in for the feature bloat.
EDIT:
> Try to limit use of the walrus operator to clean cases that reduce complexity and improve readability.
Facepalm.
But now if our big paint point is assigning a variable to len(), then in two lines evaluating that variable and printing it out, Python is just trying to find stuff to add.
I miss the size and simplicity of old Python. :/
- Complex is better than complicated.
Using the walrus operator makes the code more complex, but less complicated.
Reference: https://www.python.org/dev/peps/pep-0572/#scope-of-the-targe...
> Simple is better than complex.
I understand that nobody will force me to use this feature, but as I said before, even the spec and the write up tells how this feature is confusing and .. complex.
I know and I believe there is a purpose to be exactly after that line. Writing only simple code is not enough. If you write everything in a simple manner your code will most likely be complicated, that's exactly why you have next - complex is better than complicated.
I see this feature the same way as comprehensions, scary and complex at first, but once you learn it you would wish anyone would use it.
Python is a fairly old, mature language.
What features would you have been especially excited about?
I'm also looking forward to PEP-554 [0], which allows for "subinterpreters" for running concurrent code without removing the GIL or incurring the overhead of subprocesses.
The current extension module API encourages global state, e.g. types (and objects too) are allocated statically in a global C variable. For example, there is a global C variable `_Py_NoneStruct` that is the Python `None` value, and extension modules are accessing this variable directly.
Every use of this object needs to adjust its reference count, and that reference count is directly stored within the C global variable.
`_Py_NoneStruct` is currently even exposed in the PEP 384 stable ABI. Existing extension module binaries are commonly directly touching `_Py_NoneStruct.ob_refcnt` without any synchronization. Breaking the PEP 384 compatibility promise is fundamentally unavoidable here.
One of two things must happen: Alternative one: All refcount operations must be made atomic for thread-safety. These are really really common in the Python interpreter, but atomic operations are expensive on modern CPUs (especially if there's contention). But multiple threads using the value `None` would be quite common in Python code, so I doubt you'd gain any speed even at today's core counts -- in fact I'd expect the constant inter-CPU-communication for the refcounts to make everything slower than just using a single core with today's Python!
So alternative two, ensure no Python objects are shared between the subinterpreters. That's the plan. But that also means it's a breaking change for extension modules. And it's not just an ABI change (which would be handled by merely recompiling against the new headers). Any extension modules that do not yet support PEP 489 are already incompatible with subinterpreters, so that will take quite a bit of work until the ecosystem is upgraded. But there will probably also be some other breaking API changes. I think type objects are currently shared across subinterpreters, and those are frequently defined as a `Py_TypeObject` global variable in extension module code. Also, if every subinterpreter has its own GIL, extension modules calling `PyGILState_Ensure()` will have to specify which subinterpreter they will be using, so that the appropriate lock can be acquired.
My prediction: 3.9 may have the basic functionality, but it still won't be able to run on multiple cores concurrently. That will take a bunch of more work and breaking changes, and will likely be released as Python 4.0.
There will be another slow upgrade process ("my dependencies must upgrade before I can") until the Python ecosystem is multi-subinterpreter-compatible. But at least this one only affects extension modules.
If anything, user-input strings must be treated as tainted whatever the case.
1) existing uses of .format() would break:
aifc.py: raise Error('marker {0!r} does not exist'.format(id))
2) existing uses of "%" formatting would break: argparse.py: result = '{%s}' % ','.join(choice_strs)
3) many regular expressions would break: uuid.py: if re.fullmatch('(?:[0-9a-f][0-9a-f]-){5}[0-9a-f][0-9a-f]', value):
4) existing strings which contain uuids would break: uuid.py: >>> x = uuid.UUID('{00010203-0405-0607-0809-0a0b0c0d0e0f}')1) customizable templates (can't allow the user to access arbitrary variables)
2) i18n (can't allow the translator to access arbitrary variables)
3) complex values being passed to .format() instead of some variable
I wonder if there is anything that can really be done, outside of something like Cython.
I compile my cli tools with nuitka [0], the resulting binaries take half the time to start. I find the difference quite notable.
But startup time increases an order of magnitude once you start importing 'requests' and other common packages.
Perhaps an option to turn imports lazy...
If you use setuptools and place all your configuration declaratively in setup.cfg it is not that bad.
I've started using Docker for ensuring prod behaves the same as dev. Going from Mac dev to Linux prod would otherwise cause trouble.
Does your container expose ports or need volumes? Does it need gunicorn or uwsgi in front of it? What about system packages? None of those (except maybe the last one) are really in the scope of the package manager.
Python's heuristics for that are pretty annoying.
If it's any consolation, the scope of a variable isn't determined by heuristics, it's just that the rules are (IMO) kinda bad. Basically, if a variable is assigned (in addition to `=` and friends, `import`, `class` and `def` are also assignments in disguise) to in what Python calls a "block" (which is not like a C-style block; rather `class` body, `def` body, or top-level) it is considered to belong to that block (with the notable exception of the name for a caught exception in a `except Exception as e: ...`, see [1]). If you want it to refer to a variable outside the block, you need to use `global` or `nonlocal` as appropriate. The full gory details are in [2]
1: https://docs.python.org/3/reference/compound_stmts.html#the-...
2: https://docs.python.org/3/reference/executionmodel.html#nami...
Python didn't use to have lexical scoping, so the rules are retrofitting around that.
"end" statement which would enable automatic indentation.
"switch" statement instead of "if/elif/.../else" hell.
Correctly done multi-line lambdas/expressions could bring Python's expressivitiy to where ECMAScript is nowadays, where you can write both foo = function(...) or function foo().
Perl 6 has a great switch statement, called "given" https://docs.perl6.org/language/control#given
You can use mypy for that: https://github.com/python/mypy
Emacs Lisp, at least, can use question marks in names without batting an eyelash.
I see no reason that would not have been true for Lisps generally since their birth.
[1] http://www.lispworks.com/documentation/HyperSpec/Body/f_list...
In particular, without the walrus operator you can't write an if-elif chain of regex comparisons. The best way to that that I've been able to find is with a bunch of nested if-else statements.
https://github.com/eveem-org/panoramix
(source code for Eveem.org, which is arguably the best decompiler for Ethereum smart contracts out there. you can see a lot of pattern matching in pano/simplify.py , and I found no way to do it without extending the language/walrus while maintaining the readability)
I went to your repo, and it wasn't clear to me what Panoramix actually is or does. So I checked out Eveem.org, and even on the about page, it wasn't clear to me what Eveem was.
But with that context, coming back to your GitHub page, I was able to get the gist of it.
It might be helpful to have a section above the installation section that gives a little context / tells about the project.
And in your example section -- again, I literally don't know anything about decompilation so maybe I'm not your target demographic... but if I understand the gist of it, kitties is an example function or binary. It might be better to use something more concrete (and provide the context).
Feels like a case of preferring cleverness over readability/usability/maintainability.
To give you an example, let's look at this statement:
if exp ~ ('mask_shl', int:size, int:offset, -offset, :e): ...
It is the same as writing: if type(exp) == tuple and len(exp) == 5 and exp[0]=='mask_shl' and type(exp[1])==int aand type(exp[2])==int and exp[3] == -exp[1]: size, offset, e = exp[1], exp[2], exp[4] ...
If you can figure out syntax that makes this sort of matches more readable, I'll gladly use it :)
if exp ~ ('mask_shl', int:size, int:offset, -offset, :e): ...
vs if type(exp) == tuple and len(exp) == 5 and exp[0]=='mask_shl' and type(exp[1])==int aand type(exp[2])==int and exp[3] == -exp[1]:
size, offset, e = exp[1], exp[2], exp[4]
...Same can be said about e.g. Cobol.
Cobol also was a good idea back in the day. And some of the ideas in Cobol we know recognize as bad _had_ to be tried to gain that recognition.
There have been dozens of working C++ competitors over the years, each one technically as good as C++.
They all failed because their authors are good at writing languages, but very bad at knowing how languages are used in the real world.
The C++ committee really knows what it's doing, you're not going to do anything useful in this space without actually understanding the problem domain.
TL;DR - yes, C++ is complex, but that is because programming is complex. Your attitude of "programming is hard, let's go (language) shopping" will not displace C++.
I totally agree that C++ of 21st century is way better than C++ from 1998, and is becoming better. The problem is that maintaining backwards compatibility requires to keep a number of footguns in place, and some of them are cornerstones of the language :(
I have high hopes for Rust becoming the more sane competitor in the space traditionally occupied by C++. Its authors seem to understand the problem domain from a very practical standpoint, and are proficient in C++, to begin with. They also strive to apply reasonable design principles, and some actual math hopefully will keep the core of the language from logical pitfalls.
Another example of hitting a sweet spot is Go. I dislike many Go's language decisions. (Some are definitely great, though, e.g. the whole OOP approach lifted from Oberon.) But its authors definitely understand both the problem domain, and the target audience.
This is what I mean by talking about success despite the flaws. C++ has both flaws and merits, and merits outweighed the flaws. But the flaws are still painful.
class Any:
def __eq__(self, other):
self.value = other
return True
Then you can use an instance of it as the hole in a pattern. My use case was performing peephole optimizations in a simple 6502 assembler. For example, if the LDA operation (load value from memory location into accumulator) occurs twice in a row, the first load is redundant. So I did this test: x, y = Any(), Any()
if L[i:i+2] == [('LDA', x), ('LDA', y)]:
L[i:i+2] = [('LDA', y.value)]
(Note that the last line changes the length of the list, so it may be slow on large lists.)I was going to write a blog post about this back then, but I never did. If anyone is interested, I can write it now.
It is some spooky action at a distance.
https://github.com/eveem-org/panoramix/blob/bc55be84a9a6abb1...
It uses "~" custom operator, but implementing it would not be possible without ":=" operator (see /tilde directory).
Also, I wonder if there's an easy way to implement this in a jupyter notebook, without fiddling with the kernel...
(issues of whether it's a good idea aside, definitely seems useful to know about.. )
Also, in my case I had to use Codecs because a new operator causes AST-parser to not go through. But if you have some kind of syntax that will be compatible with Python AST, then you can implement it even easier.
You can google Python AST to get a lot of useful information :)
$ perl
$a = "foo";
if ( my $a = "bar" ) { print "$a\n" }
print "$a\n"
bar
foo if (ptr = strchr(address, ':'))
port = atoi(ptr + 1);I love the walrus operator with a love that is unholy.
(although I think "else" should have been a different keyword)
Agreed.
There's a number of (full) languages that compile to the Python AST [0] (I'm especially fond of http://hylang.org), but they're very different from Python. It would be interesting to see smaller variations of standard Python implemented in an interoperable way like these languages do.
[0] https://github.com/vindarel/languages-that-compile-to-python
I did not follow all the conversation around the typing system but isn't the whole point of it to propose optimizations in the future? I find it exciting
I also thought, why bother at first until I learned this is already a feature in python, but only for c-functions. So of course it makes sense to level the playing field.
Most devs don't understand that software can be done and still be useful.
Like how they removed the `print` keyword when going from Python 2 to Python 3.
But removing is harder than adding.
The "walrus operator" will occasionally be useful, but I doubt I will find many effective uses for it. Same with the forced positional/keyword arguments and the "self-documenting" f-string expressions. Even when they have a use, it's usually just to save one line of code or a few extra characters.
The labeled breaks and continues proposed in PEP-3136 [0] also wouldn't be used very frequently, but they would at least eliminate multiple lines of code and reduce complexity.
PEP-3136 was rejected because "code so complicated to require this feature is very rare". I can understand a stance like that. Over complicating a language with rarely-used features can definitely create problems. I just don't see why the three "headline" features I mentioned are any different.
[0]: https://www.python.org/dev/peps/pep-3136/