tasks = {}
for url in URLs:
future = executor.submit(fetch_url, url)
tasks[future] = url
bothers you, this is perfectly (some would say more so even than the original) Pythonic: tasks = {executor.submit(fetch_url, url): url for url in URLs} def main():
with ThreadPoolExecutor(max_workers=len(URLs)) as executor:
for url, title in zip(URLs, executor.map(fetch_url, URLs)):
print(f"URL: {url}\nTitle: {title}")
The default value of `max_workers` since
Python 3.8 has been min(32, os.cpu_count() + 4)
It seems like you should avoid max_workers=len(items_to_process)
It won't save resources with smaller collections of items (workers are created when needed) and may waste resources with larger.Is reading the official docs section on concurrency lots of research?
The amusing part is that the article calls out two groups of people into which your advice falls.
It’s not that much code, it’s about 4 lines of code, creating a “pool” and calling a wait on future objects.
This is a perfect solution for Python developers who have been perfectly happy using Django for years, and just need to scrape some API or download multiple files.
No, they shouldn’t switch to a different language the moment they need to optimize something embarrassingly parallel, they can see whether a simple solution in stdlib is enough, and probably move on.
Yes, I've written a shit ton of code in all aforementioned languages.
[1] https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-s...
It happened with me and many other former colleagues.
Though obviously, everyone decides for themselves when does that point come -- or if it comes at all.