Show HN: A Python Script to Download Thousands of Wallpapers at Once

Re:Show HN: A Python Script to Download Thousands of Wallpap... by myroon5 on Monday April 23, 2018 @12:22PM (#16904820)

It looks like this downloads them one-by-one instead of downloading at the same time.

Is there a simple Python equivalent to Java's .parallelStream().forEach() that would allow these calls to easily be run in parallel?

[ Parent | on HN ]

Re:Show HN: A Python Script to Download Thousands of Wallpap... by mynewtb on Monday April 23, 2018 @12:26PM (#16904855)

Sure thing...

    from multiprocessing import Pool
    with Pool(8) as p:
        p.map(function, sequence))

[ Parent | on HN ]

Re:Show HN: A Python Script to Download Thousands of Wallpap... by myroon5 on Monday April 23, 2018 @02:31PM (#16906110)

Neat.

I see that you explicitly specify 8 as the number of processes here. In Java, parallelStream() will pick a sane default for you if you haven't previously specified (based on the number of available processors, I believe). Is something like that possible in Python?

[ Parent | on HN ]

Re:Show HN: A Python Script to Download Thousands of Wallpap... by jstarfish on Monday April 23, 2018 @02:38PM (#16906184)

The ThreadPool class picks a sane default (number of cores), but I believe it uses Python threads instead of processes.

[ Parent | on HN ]

Re:Show HN: A Python Script to Download Thousands of Wallpap... by pddubs on Monday April 23, 2018 @12:33PM (#16904944)

You could just use threads here too - CPython releases the GIL during IO.

  from concurrent.futures import ThreadPoolExecutor
  with ThreadPoolExecutor as executor:
      executor.map(function, sequence)

[ Parent | on HN ]