News for hackers. Stuff that matters.
Re:The easy way to concurrency and parallelism with Python s... by pdimitar on Thursday September 14, 2023 @07:35AM (#37507569)
Certainly. My point is that if you need to write that much code and/or do that much research, at one point the effort of doing it in another language will be less than to keep insisting on using a tool that's not designed for it.

It happened with me and many other former colleagues.

Though obviously, everyone decides for themselves when does that point come -- or if it comes at all.

Re:The easy way to concurrency and parallelism with Python s... by FreakLegion on Thursday September 14, 2023 @07:46AM (#37507645)
The point of the article is a handful of lines. The rest is accoutrement like the URL list and timing code. But sure, if

    tasks = {}
    for url in URLs:
        future = executor.submit(fetch_url, url)
        tasks[future] = url
bothers you, this is perfectly (some would say more so even than the original) Pythonic:

    tasks = {executor.submit(fetch_url, url): url for url in URLs}
Re:The easy way to concurrency and parallelism with Python s... by throwaway46281 on Thursday September 14, 2023 @09:21AM (#37508660)
As a side note, using a future as a map key struck be as a bit weird, though perfectly valid. It'd be more natural IMO to use a list for the futures, and have the fetch_url function return a (url, result) tuple. Or use the url as the map key and just iterate over the map items instead of using as_completed on the keys
Re:The easy way to concurrency and parallelism with Python s... by networked on Thursday September 14, 2023 @01:50PM (#37512347)
I have looked through the documentation for `concurrent.futures`, and there is another option. You can use `Executor.map` (https://docs.python.org/3/library/concurrent.futures.html#co...). It eliminates the need to wait on the futures explicitly.

  def main():
      with ThreadPoolExecutor(max_workers=len(URLs)) as executor:
          for url, title in zip(URLs, executor.map(fetch_url, URLs)):
              print(f"URL: {url}\nTitle: {title}")
The default value of `max_workers` since Python 3.8 has been

  min(32, os.cpu_count() + 4)
It seems like you should avoid

  max_workers=len(items_to_process)
It won't save resources with smaller collections of items (workers are created when needed) and may waste resources with larger.
Re:The easy way to concurrency and parallelism with Python s... by pid-1 on Thursday September 14, 2023 @07:58AM (#37507737)
> and/or do that much research,

Is reading the official docs section on concurrency lots of research?

Re:The easy way to concurrency and parallelism with Python s... by Demiurge on Thursday September 14, 2023 @08:25AM (#37507977)
What “much research” are you talking about?

The amusing part is that the article calls out two groups of people into which your advice falls.

It’s not that much code, it’s about 4 lines of code, creating a “pool” and calling a wait on future objects.

This is a perfect solution for Python developers who have been perfectly happy using Django for years, and just need to scrape some API or download multiple files.

No, they shouldn’t switch to a different language the moment they need to optimize something embarrassingly parallel, they can see whether a simple solution in stdlib is enough, and probably move on.

Re:The easy way to concurrency and parallelism with Python s... by oefrha on Thursday September 14, 2023 @09:56AM (#37509092)
If this is too much research for you, wait until you have to deal with the many problems of Go channels in the real world. (Reasonably well-known though controversial article: [1]) Don't even get me started on Rust. Concurrency and parallelism is hard.

Yes, I've written a shit ton of code in all aforementioned languages.

[1] https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-s...