With what I learned about Go and concurrency, I would say that currently in Python, writing concurrent code is not very hard, and is as close to Go as you can get without actually just writing Go.
Now, you may be saying "but Python has the GIL, how can concurrency be easy in Python?" I'd say, you're definitely not wrong that the GIL is a problem, but it's not much of a problem for concurrency.
This goes back to the heart of Rob Pike's classic talk, "Concurrency Is Not Parallelism"[2]. To quote Wikipedia:
In computer science, concurrency is the decomposability property of a
program, algorithm, or problem into order-independent or partially-ordered
components or units.
In closing, concurrency in Python is pretty easy to work with, it just performs very poorly.
[1] - https://github.com/lelandbatey/defuse_division
[2] - https://blog.golang.org/concurrency-is-not-parallelism
https://github.com/rcarmo/python-utils/blob/master/taskkit.p...
Obviously, it wasn't amazingly performant. But it did help a lot for doing concurrent stuff, and I've been pondering re-doing it for asyncio.
In principle it's possible to implement something like JyNI (http://jyni.org/) or CPyExt (https://morepypy.blogspot.de/2010/04/using-cpython-extension...) to bridge the CPython and Grumpy APIs. In practice, marshalling data across the interface can be very expensive.
If this is good enough to run YouTube's python code already it's honestly super impressive. Well done.
There are a handful of C extensions for JSON, protobufs, etc that YouTube uses, but mostly they're small utility functions written by us to optimize particularly hot code paths.
(If the former, you could just update the code to use the standard Go JSON/etc packages..)
Also, further signs that GC may not be the reason, is that D also has GC, but can link to C libraries somewhat easily (not sure about all cases or how far the ease goes).
The alternate stack structure is indeed one issue. The bigger one is the GC, though; the Go runtime needs to know which pointers it is responsible for freeing, and which are the responsibility of the C code.
That is not the bigger issue, and AFAIK already handled for C types.
The stack/calling conventions is the reason why cgo is "not go", cgo calls have significantly more overhead than just about every other FFI (the overhead of a cgo call is ~2 orders of magnitude more than a "native" go call, or was around the same time last year, that is you could perform ~100 no-op non-inlined native calls to a do-nothing function by the time you need for a single cgo call to the same).
Interesting, didn't know this (that Go code runs in an event loop). Is the reason something to do with goroutines and channels? something like, a routine gets info that data is available for it to read (on a channel, sent by another goroutine), via an event it receives?
Also, can you explain this point:
"which enables excellent I/O performance without kernel context-switches" ?
Before you can call any coroutine you first need to start an event loop and schedule something in it. This essentially enables the language to schedule another async function each time you use await.
Since Go by default always is async, before your main function is called, it sets up the even loop and then calls your main, which technically is also a coroutine. Your code appears to be sequential, but it is not executed that way.
Grumpy likely doesn't support the C extensions due to time, and complexity of having to actually emulate the GIL since Python does not have fine grained locking for structures. C extensions that work with Python data structures need to first hold the GIL.
It's because Python's C API is inherently non-thread safe. The API lacks passing an interpreter pointer as a parameter (as Lua's API does for example). So Python is forced to use a terrible thread local storage hack involving the Global Interpreter Lock to swap interpreter instances which is insanely inefficient and limits compute-bound programs to a single thread.
Python 3.x had a chance to fix the API and do away with the GIL once and for all, but inexplicably they did not. There was a misguided notion that C extensions between 2.x and 3.x could be interoperable.
My experience with Tcl teached me to stay away from languages that don't have either a JIT or AOT compiler on their reference implementation.
Just like people learned 8-bit BASIC and went on to do business applications and games on it. I went Z80 ASM instead.
Personally I would only use Python for shell scripting and advise for using Julia instead.
Of course, others see it differently.
Do you have real examples here or is this just FUD? SciPy is not known for using absolute state of the art algorithms or perfectly optimized implementations. It can be pretty easy to improve on the naively implemented or legacy pieces of SciPy, e.g. http://tullo.ch/articles/python-vs-julia/
> SciPy, which, you may not be aware, is basically C and Fortran code wrapped in Python API.
Which for me personally means, that I would rather C and Fortran directly or better yet, a C++, .NET or Java binding to them.
Being able to describe things with a syntax that looks almost like pseudocode and runs highly-optimized C/Fortran code to do heavy lifting has huge, huge advantages.
Also Java and .NET ecosystems are just a little bigger than Python.
LANPACK + BLAS = scipy
> Numpy/Scipy are only relevant to a minor set of computer users..
> ... Java and .NET ...
Being able to describe things with a syntax that looks almost like pseudocode and runs highly-optimized C/Fortran code to do heavy lifting has huge, huge advantages.
Thanks, I already knew that.
> Being able to describe things with a syntax that looks almost like pseudocode and runs highly-optimized C/Fortran code to do heavy lifting has huge, huge advantages.
Hence we are back at Scala, Clojure, F#, enjoying the respective AOT/JIT native code compilers, and integrating with that highly-optimized C/Fortran code.
Functional languages. The "year of Linux on the desktop" of programming languages. Also none of those look like pseudocode (Scala does if you ignore bits and squint).
Other sites have similar totals. But they are filtering for London/UK/Europe, so that might be a bias.
It's interesting to see the same pain has now made caused the runtime itself to be implemented in Go.
It's a pity C extensions (often used in scientific computing) are not supported but Go does have support via CGO, so maybe some approach can be worked out to access C routines in the future.