Memory efficiency of parallel IO operations in Python

Re:Memory efficiency of parallel IO operations in Python by akvadrako on Tuesday March 13, 2018 @12:47PM (#16577506)

I guess you are right - I haven't been able to find the sources I was remembering that gave me this idea. Maybe it was just about pid_max. The default of about 32K clearly wouldn't cut it.

The max is only 4M so I guess I'll change my statement to be linux doesn't handle millions of threads very well.

[ Parent | on HN ]

Re:Memory efficiency of parallel IO operations in Python by dekhn on Tuesday March 13, 2018 @12:56PM (#16577572)

You're probably thinking of Linux's older thread system, LinuxThreads, which didn't scale at all. It was replaced with NPTL, which scales far better. I think also the kernel scheduler got much better.

I'm not aware of any system that manages millions of threads well- it's also not clear what the usecase for this would be.

[ Parent | on HN ]

Re:Memory efficiency of parallel IO operations in Python by akvadrako on Tuesday March 13, 2018 @01:04PM (#16577629)

I'm not thinking of LinuxThreads, but internally Linux still treats threads and processes as one. So each thread is assigned a PID.

As per the top post, millions of threads means there is no need to implement your own scheduling system with async IO, which can handle millions of idle connections.

[ Parent | on HN ]