News for hackers. Stuff that matters.
Re:Memory efficiency of parallel IO operations in Python by zzzeek on Tuesday March 13, 2018 @08:58AM (#16575729)
I have an unpopular opinion. Non blocking IO is useful when we want to scale to lots and lots of parallel IO channels. If we don't have lots, only dozens, there is no strong need for non blocking IO. You don't actually need anything like coroutones or event driven frameworks to use non blocking IO, just loop through the sockets and tend to the ones that are ready, though event driven frameworks can make it much nicer (though still significantly more awkward than writing code in a synchronous paradigm).

Within the world of "we want non blocking IO", the world has gone crazy. JavaScript has created a whole generation of programmers who think inside out callback / yielded code is how everything should be done, and that threads are "too hard". They hardly understand the original purpose of non blocking IO and conflate the fact that they can conceptualize event driven code better than they can threads with "well that's why it's so much faster" (which usually it isn't). I can't retire fast enough from this industry.

Re:Memory efficiency of parallel IO operations in Python by quentinp on Tuesday March 13, 2018 @01:13PM (#16577706)
I think many people agree that speed is not a reason to choose non-blocking IO over threads.

However, many people do believe that async/await and event loops make reasoning about non-blocking IO much easier. Has your opinion changed since http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...?

Re:Memory efficiency of parallel IO operations in Python by zzzeek on Tuesday March 13, 2018 @04:28PM (#16579496)
> However, many people do believe that async/await and event loops make reasoning about non-blocking IO much easier.

they do, until their program is mysteriously having MySQL server drop their connections randomly because something CPU-bound snuck in and the server dumps non-authenticated connections after ten seconds. Three days of acquiring and poring over HAProxy debug logs from the production system finally reveals the issue that never really should have happened in the first place because the server is only handling about 30 requests per second, and of course the fix is to switch that part of the program to threads.

asyncio certainly makes it easier to reason about non-blocking IO but it also means you have to construct your own preemptive multitasking system by hand, given only points of IO where context can actually switch. We're coding in high level scripting languages. Low level details like memory allocation, garbage collection, and multitasking should be taken care of for us.

Re:Memory efficiency of parallel IO operations in Python by quentinp on Wednesday March 14, 2018 @12:33AM (#16582364)
Threads are a leaky abstraction, so it makes sense to explore other solutions. To cite another low level detail from you list: I don't think Rust is less usable because there is no garbage collection.

But keep in mind asyncio has many issues of its own, which is why I'm happy that alternatives like http://trio.readthedocs.io/ are possible in Python.

Re:Memory efficiency of parallel IO operations in Python by ben0x539 on Tuesday March 13, 2018 @04:21PM (#16579431)
I don't think that's such an unpopular opinion. At least I think I'm mostly on board. :)

I don't really want to write callback-driven code anywhere, but even for simple definitely-not-thousands-of-concurrent-connections kinda problems, the typical language's standard library operation of "do this one thing and wait indefinitely until it's done" gets fairly annoying, working around it with threads is also annoying, and I'm just kinda hoping that the people working on making the async IO user experience better will accidentally make my life easier as well.

Re:Memory efficiency of parallel IO operations in Python by djd20 on Wednesday March 14, 2018 @03:13AM (#16582921)
I suspect many people would be very sad if you retired from the industry! Am pretty sure I would quit if sqlalchemy stopped being developed.