Re:Async Python is not faster
by heavyset_go on Friday June 12, 2020 @06:30PM
(#23504540)
Each Linux thread has at least an 8MB virtual memory overhead. I just tested it, and was able to create one million coroutines in a few seconds and with a few hundred megabytes of overhead in Python. If I created just one thousand threads, it would take possibly 8 gigs of memory.
[ Parent
| on HN ]
Re:Async Python is not faster
by otabdeveloper4 on Saturday June 13, 2020 @03:29AM
(#23507586)
Virtual memory is not memory. You're effectively just bumping an offset, there's no actual allocations involved.
> ...it would take possibly 8 gigs of memory.
No. Nothing is 'taken' when virtual memory is requested.
But have you tried creating one thousand of OS threads and measuring the actual memory usage? If I recall correctly I read some article where it was explained that threads in Linux are not actually claiming their 8MB each so literally. I need to recheck that later.
Re:Async Python is not faster
by heavyset_go on Friday June 12, 2020 @11:45PM
(#23506585)
You're right, I've read the same. Using Python 3.8, creating 12,000 threads with `time.sleep` as the target clocks in at 200MB residential memory.