A couple more along the same lines:
- Metaclasses and type. (This is admittedly dark magic, but useful in library code, less so in application code)
- Magic methods! Everyone knows about __init__, but you can override all sorts of behaviors (see: https://docs.python.org/3/reference/datamodel.html)
My favorite example (I have a lot of favorite examples :)) is __call__, which emulates function calling and is the equivalent of C++'s operator().
Why is it my favorite? Because as the old adage goes, "a class is a poor man's closure, a closure is a poor man's class":
class C:
def __init__(self, x):
self.x = x
def __call__(self, y):
return self.x + y
>>> a = C(2)
>>> a(3)
5I didn't have to use Metaclasses, either, though I have read about them, especially in Fluent Python. But I guess I belong to the 99% who haven't had to worry about them, yet :P
What is the benefit compared to having a method named "add" that also explains the behavior?
It may also give you a "clearer" (in quotes because subjective) presentation for something you're trying to do.
processor = SomeProcessor.load("path/to/config")
# with __call__
processed_inputs = processor(inputs)
# less awkward than
processes_inputs = processor.process(inputs)
If you don't know it already, it is really worth looking into. I am a python dev with nearly a decade of experience and I knew generators, and yet this was still an eye opener.
The concepts matter more than the chosen language in this deck.
I learned a lot! Looks like I can apply this to a PHP trace/profile parser project, especially the pipelined parsing and the query language idea.
is ps internal variable unique for each Thread or same?
is it safe to execute your primes() from different threads?
If you run it from different threads I guess it will be the same as calling the function multiple times, it will return a new started-from-the-top generator.
def sum():
yield 1
yield 2
print(repr(sum()))
print(next(sum()))
print(next(sum()))
<generator object sum at 0x7fc6f14823c0>
1
1You'll need to search, or try!
If you try to send g to a different process, you will get an error, because it doesn't serialize.
The best way to think of a generator is as an object implementing the iteration protocol. They don't really interact with concurrency, as far as multiprocess is concerned, they're just regular objects. So the answer is that it depends on how you plan to share memory between the processes.
> is ps internal variable unique for each Thread or same?
ps is local to the generator instance.
def f():
x = 0
while True:
yield (x := x + 1)
>>> f()
<generator object f at 0x10412e500>
>>> x = f()
>>> y = f()
>>> next(x)
1
>>> next(x)
2
>>> next(y)
1
For this specific generator, you would run into the GIL. More generally, if you're talking about non CPU-bound operations, you need to synchronize the threads. It's worth looking into asyncio for those use cases.
https://stackoverflow.com/questions/20579756/passing-value-t...
And don't forget "yield from" (same as yielding all values in a list, but keeps the original generator! You can send data back to the list if it is itself another generator!)
[
{"name": "/dev/loop0"},
{"name": "/dev/loop1"},
{"name": "/dev/loop2"},
{
"name": "/dev/sda",
"children":
[
{
"name": "/dev/sda1",
"children":
[{"name": "/dev/mapper/lubuntu--vg-root"}, {"name": "/dev/mapper/lubuntu--vg-swap_1"}],
},
],
},
{"name": "/dev/sdb", "children": [{"name": "/dev/sdb1"}, {"name": "/dev/sdb2"}]},
{"name": "/dev/sdc", "children": [{"name": "/dev/sdc1"}, {"name": "/dev/sdc9"}]},
]
def flatten(items):
for item in items:
yield {k:v for k,v in item.items() if k != 'children'}
if 'children' in item:
yield from flatten(item['children'])
[{'name': '/dev/loop0'},
{'name': '/dev/loop1'},
{'name': '/dev/loop2'},
{'name': '/dev/sda'},
{'name': '/dev/sda1'},
{'name': '/dev/mapper/lubuntu--vg-root'},
{'name': '/dev/mapper/lubuntu--vg-swap_1'},
{'name': '/dev/sdb'},
{'name': '/dev/sdb1'},
{'name': '/dev/sdb2'},
{'name': '/dev/sdc'},
{'name': '/dev/sdc1'},
{'name': '/dev/sdc9'}] def flatten(children=[], **other):
if other: yield other
for child in children: yield from flatten(**child) question_bank={'1+1' : '2', '2+3' : '5'}
def Quiz():
for question, correct_answer in question_bank.items():
answer = yield question
if answer == correct_answer:
print('Correct!')
else:
print('Wrong.')
yield 'Finished!'
question = Quiz()
q = next(question)
while q != 'Finished!':
q = question.send(input(q)) def get_api_results(query):
params = { "next_token": None }
while True:
response = requests.get(URL, params=params)
json = response.json()
yield from json["results"]
if json["next_token"] is None:
return
params["next_token"] = json["next_token"]
for result in get_api_results(QUERY):
process_result(result) # No need to worry about paginationI think you could simplify the rest like so:
def primesHN():
from collections import defaultdict
from itertools import count
yield(2)
ps = defaultdict(list)
for i in count(3,2):
if i not in ps:
yield(i)
ps[i**2].append(2*i)
else:
for n in ps.pop(i):
ps[i + n].append(n)It is. Itertools is a masterpiece of a module. It has a lot of functions that operate on iterators and will work both on standard iterables (lists, tuples, dicts, range(), count() etc.) and on your own generators. It forms a sort of "iterator algebra" that makes working with them very easy.
> I think you could simplify the rest like so:
Sounds good, but with a caveat: you do need to call "del" at the end for memory deallocation purposes. The garbage collector isn't smart enough to know you won't be using those dictionary entries any longer. Technically the code still works, but keeping everything in memory defeats the purpose of writing a generator.
Using "yield" instead of "return" turns the function into a coroutine. This is useful in all sorts of cases and works very well with the itertools module of the standard library.
One of my favorite examples: a very concise snippet of code that generates all primes: