hoodchatham
hoodchatham on Hacker News
> Pyodide and Cloudflare don't use the real network stack for http requests
Pyodide in browsers _can't_ use the real network stack because of fundamental security principles of browsers. With direct networking you could get around the CORS restrictions.
Thanks to Gyeongjae Choi's work, Pyodide in Node/Cloudflare can use direct sockets. See this section of the blog post:
https://blog.cloudflare.com/python-workers-ga/#using-postgre...
> JS event loop is preemptive
I think the word you want is "eager". Preemptive usually refers to things like signal handlers: when the signal is received the handler "preempts" normal execution without waiting for an explicit yield point.
In any case, with the WebLoop, Python coroutines stay lazy. The fundamental primitive a Python event loop needs to implement is call_later() which maps fairly cleanly to `setTimeout()`.
The reason we want to use the JS event loop is that the JS event loop is where all the actual I/O events in a JavaScript runtime happen. If you create a second event loop and run it, it will block actual IO on the JS event loop. So making uvloop work would be pointless.
In practice it is easier to support rust packages on web assembly than C or C++ packages since Rust cross compilation is in better shape. Of course packages that do IO can be a challenge but problematic IO is also more common in C libraries than Rust libraries.
Tokio doesn't work yet but epoll-based support for it is in progress.
crates.io is for distributing source code, pypi is primarily for distributing wheels which contain a mixture of Python code and compiled shared libraries.
That video was a major inspiration for me and probably indirectly led to me becoming a Pyodide maintainer.
It says the same thing in the pyodide repl for what it's worth.
Reference types makes wasm/js interoperability way cleaner and easier. wasm-gc added a way to test a function pointer for whether it will trap or not.
And JSPI is a standard since April and available in Chrome >= 137. I think JSPI is the greatest step forward for webassembly in the browser ever. Just need Firefox and Safari to implement it...
The actual code example he made runs about 100 times faster in Spy/emscripten than in Pyodide.
Pyodide uses its own event loop which just subscribes to the JavaScript event loop. My suspicion is that this will be more efficient than using uvloop since v8's event loop is quite well optimized. It also allows us to await JavaScript thenables from Python and Python awaitables from JavaScript, whereas I would be worried about how this behaves with separate event loops. Also, porting uvloop would probably be hard.
Are people maintaining wasi ports of Python 2.7 and 3.8?
I certainly think there's an 80/20 rule here. Most packages are not very hard to port, and generally the ones that are hard to build use features like threads and multiprocessing, graphics cards, raw sockets, green threads, or other capabilities that have no obvious analogue in a webassembly runtime.
As we mention in the blog post, the biggest issues are around supporting server and request packages since they are clearly useful in cloudflare workers but are difficult to port because they frequently use raw sockets and some form of concurrency.
As far as I'm aware, even discarding the instance isn't good enough, since v8 doesn't seem to reclaim the Wasm Linear Memory ever. I think the only thing you can do is start it in a worker and then terminate the entire worker.
To be clear these are all use cases of Pyodide. PyScript only was released two days ago so it's harder to point to real world uses.
> Why?
This isn't good for most web use cases but there are some major new use cases that it enables.
Pyodide is in Python documentation all over the place, for instance on https://numpy.org/
Python has many libraries that JS is missing. Particularly the scientific computing ecosystem has a lot of very well supported Python packages but fewer JavaScript packages. They also can benefit a lot from connection to a UI.
Two key use cases for this stuff are Python education and scientific computing. Students have a hard time installing Python locally. Generally scientists want to share their results broadly, but they probably did their analysis in Python.
Here's another example of a good use case:
https://www.socialfinance.org.uk/blogs/analysing-sensitive-d...
Adding -Os to the Pyodide compilation flags has proven surprisingly hard:
https://github.com/pyodide/pyodide/pull/1882But some other low hanging fruit include unvendoring the special encodings for Asian languages (hopefully everyone uses utf8), the decimal library, and the xml library which are all quite large and only occasionally used.
Agreed, wasm loading is supposed to be cached but in practice it doesn't seem to happen.
I'm confused how this would work. What about METH_VARARGS | METH_KEYWORDS functions? These are declared with type PyObject* f(PyObject*, PyObject*, PyObject*), are cast to PyObject *f(PyObject*, PyObject*) for the PyMethodDef, and then are cast back to their original correct type before being called. Wouldn't your approach generate bad shims for these?