The easy way to concurrency and parallelism with Python stdlib

Re:The easy way to concurrency and parallelism with Python s... by FreakLegion on Thursday September 14, 2023 @07:46AM (#37507645)

The point of the article is a handful of lines. The rest is accoutrement like the URL list and timing code. But sure, if

    tasks = {}
    for url in URLs:
        future = executor.submit(fetch_url, url)
        tasks[future] = url

bothers you, this is perfectly (some would say more so even than the original) Pythonic:

    tasks = {executor.submit(fetch_url, url): url for url in URLs}

[ Parent | on HN ]

Re:The easy way to concurrency and parallelism with Python s... by throwaway46281 on Thursday September 14, 2023 @09:21AM (#37508660)

As a side note, using a future as a map key struck be as a bit weird, though perfectly valid. It'd be more natural IMO to use a list for the futures, and have the fetch_url function return a (url, result) tuple. Or use the url as the map key and just iterate over the map items instead of using as_completed on the keys

[ Parent | on HN ]

Re:The easy way to concurrency and parallelism with Python s... by networked on Thursday September 14, 2023 @01:50PM (#37512347)

I have looked through the documentation for `concurrent.futures`, and there is another option. You can use `Executor.map` (https://docs.python.org/3/library/concurrent.futures.html#co...). It eliminates the need to wait on the futures explicitly.

  def main():
      with ThreadPoolExecutor(max_workers=len(URLs)) as executor:
          for url, title in zip(URLs, executor.map(fetch_url, URLs)):
              print(f"URL: {url}\nTitle: {title}")

The default value of `max_workers` since Python 3.8 has been

  min(32, os.cpu_count() + 4)

It seems like you should avoid

  max_workers=len(items_to_process)

It won't save resources with smaller collections of items (workers are created when needed) and may waste resources with larger.

[ Parent | on HN ]