A Hitchhiker’s Guide to Asynchronous Programming

Re:A Hitchhiker’s Guide to Asynchronous Programming by BiteCode_dev on Saturday February 29, 2020 @09:36AM (#22450739)

I get that this guide tries to ease you in into low level concurrency concepts.

However, if you try to just get work done in Python, this is not what you want.

Don't do threads/processes yourself, use pools:

    import random
    import time
    from concurrent.futures import ProcessPoolExecutor, as_completed

    def hello():
        seconds = random.randint(0, 5)
        print(f"Start blocking for {seconds}s")
        time.sleep(seconds)
        print(f"Stopped blocking after {seconds}s")
        return seconds

    if __name__ == "__main__":

        with ProcessPoolExecutor(max_workers=2) as exec:

            a = exec.submit(hello)
            b = exec.submit(hello)

            for future in as_completed((a, b)):
                print(future.result())

And don't manage the loop yourself. Use Python 3.7, and replace:

    loop = asyncio.get_event_loop()
    loop.run_until_complete(loop.create_task(foo()))
    loop.close()

With: asyncio.run(foo())

The code is not just shorter, it is way, wayyyyyyyyyyyy, more correct.

Also don't program asyncio by hand. Use a lib. E.G: wanna do http, use aiohttp.

This is Python, don't suffer more than you need to.

[ Parent | on HN ]

Re:A Hitchhiker’s Guide to Asynchronous Programming by crazyguitar on Saturday February 29, 2020 @11:24AM (#22451308)

I agree with you. We should use a reliable library, as you said. The primary purpose of this article is to help people understand what a coroutine and an event loop are. Therefore, programmers can use asyncio API fluently without misuse.

However, I don't think use threads/processes is a bad idea. A pool gives you a constrain to utilize threads/processes, but sometime we may want to adjust the number of threads/processes based on system load. Under this circumstance, using a pool is not the best choice.

[ Parent | on HN ]

Re:A Hitchhiker’s Guide to Asynchronous Programming by heavenlyblue on Saturday February 29, 2020 @04:53PM (#22453394)

You’re sending the poor newbie on a journey of self-discovering of picklable-non-picklable, passing of arguments back and forth, working with Queues (which have well-known yet undocumented race conditions), missing exception stack traces due to dead processes and all sorts of useless garbage they don’t need to know about.

Also what exactly would that newbie be building that starts and stops threads depending on the system load? What kind of a contraption is that? What are you doing?

Finallizing all of the above: under the circumstance you mentioned, you should check whether you have just seriously over-architected the solution.

[ Parent | on HN ]

Re:A Hitchhiker’s Guide to Asynchronous Programming by crazyguitar on Saturday February 29, 2020 @07:16PM (#22454193)

I understand you are worried about newbies misuse APIs. You remind me that I should add a warning to inform the sample code in this article should not use in programs. Thanks.

Also, I did not advocate a newbie should start and stop threads by themselves. I want to say I agree that we should use high-level APIs in most cases, but, in some cases, we may need to use low-level APIs to achieve our missions. I am unwilling to limit what kind of APIs should use. In my opinion, like you said: "you have just seriously over-architected the solution," we should be careful to use APIs. Even though high-level APIs are safer, programmers may misuse them.

[ Parent | on HN ]

Re:A Hitchhiker’s Guide to Asynchronous Programming by BiteCode_dev on Saturday February 29, 2020 @02:35PM (#22452499)

Just create a new pool and close the previous one, depending of the system load.

[ Parent | on HN ]

Re:A Hitchhiker’s Guide to Asynchronous Programming by crazyguitar on Saturday February 29, 2020 @06:52PM (#22454085)

I think this is not an excellent solution. You have to wait for all threads finish. I agree that we should use high-level API in most of the time, but, in some cases, we still need low-level APIs to support us to reach some goals.

[ Parent | on HN ]

Re:A Hitchhiker’s Guide to Asynchronous Programming by skrtskrt on Saturday February 29, 2020 @05:17PM (#22453563)

+1 for this, Aio libs make all of this so effortless - aiohttp, aiopg, aiomysql, aiobotocore, I’m about to try aiokafka next.

Just an effortless programming experience.

[ Parent | on HN ]