News for hackers. Stuff that matters.
Re:Why do we need Flask, Celery, and Redis? (2019) by eloff on Saturday April 18, 2020 @10:16AM (#22907891)
Gevent is like goroutines with GOMAXPROCS=1. Which is to say not nearly as useful. It gives you concurrency without parrallelism, because Python never did shake the GIL. Which goes to show some technical debt will haunt you forever.
Re:Why do we need Flask, Celery, and Redis? (2019) by sandGorgon on Saturday April 18, 2020 @12:34PM (#22908828)
Funny you mention the GIL - this is being done as part of PEP 554 which was slated to release in python 3.9 (in alpha).

And it's showing great promise ...while it could be delayed.

https://www.mail-archive.com/[email protected]/msg108063...

2021 is probable going to be the "Gone Gil" moment !

Re:Why do we need Flask, Celery, and Redis? (2019) by amelius on Saturday April 18, 2020 @02:23PM (#22909875)
Why don't they use/fork the GoLang runtime?
Re:Why do we need Flask, Celery, and Redis? (2019) by YawningAngel on Saturday April 18, 2020 @07:23PM (#22911973)
There are a few reasons: 1. This would definitely break the CPython API, which is not an option for mainstream Python. 2. The Golang runtime isn't really well-understood as a backend for languages that aren't Golang, although I acknowledge that there's no reason in principle why you couldn't compile $LANGUAGE to Golang.
Re:Why do we need Flask, Celery, and Redis? (2019) by antoinealb on Saturday April 18, 2020 @11:14AM (#22908283)
To nuance your comment, you can still get some form of parallelism, "just not" thread parallelism in Python. You can still spawn multiple handler process, or have threaded code in a C extension.
Re:Why do we need Flask, Celery, and Redis? (2019) by eloff on Saturday April 18, 2020 @12:40PM (#22908881)
Yes, you can use multiple processes to get the parallelism. But that's quite limiting compared with goroutines. Passing data back and forth is hard, and you can petty much forget about shared data structures. Memory usage is also much higher.
Re:Why do we need Flask, Celery, and Redis? (2019) by zepto on Saturday April 18, 2020 @11:35AM (#22908421)
It’s arguable whether either of those are ‘in python’.
Re:Why do we need Flask, Celery, and Redis? (2019) by quacker on Saturday April 18, 2020 @11:50AM (#22908518)
The multiprocessing module in the standard library is absolutely a Python-native way to do parallelism:

    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))
This runs f(1), f(2), and f(3) in parallel, using a pool of five processes.

https://docs.python.org/3.6/library/multiprocessing.html

Re:Why do we need Flask, Celery, and Redis? (2019) by goostavos on Saturday April 18, 2020 @12:51PM (#22908995)
Whoa! It never occurred to me that `Pool` could be used with context manager. I've always typed out `.close()` manually like a sucker.
Re:Why do we need Flask, Celery, and Redis? (2019) by kbutler on Saturday April 18, 2020 @11:59AM (#22908567)
https://docs.python.org/2/library/multiprocessing.html

Python standard library since 2.6. Pretty much the definition of "in python".

Re:Why do we need Flask, Celery, and Redis? (2019) by quacker on Saturday April 18, 2020 @11:46AM (#22908492)
> Gevent is like goroutines with GOMAXPROCS=1.

On its own, yes. For webapps, you can easily combine it with a multi-process WSGI server (like gunicorn or similar).

Re:Why do we need Flask, Celery, and Redis? (2019) by throwaway894345 on Saturday April 18, 2020 @05:09PM (#22911120)
Running four processes with GOMAXPROCS=1 is strictly worse than 1 process with GOMAXPROCS=4. The difference is coarse-vs-fine grained parallelism. Notably, a WSGI+gevent system doesn't allow you to do parallelism within a request, not to mention configuring these WSGI implementations (especially for production) is a bunch of extra headache for which there is no analogy in Go.
Re:Why do we need Flask, Celery, and Redis? (2019) by marcosdumay on Saturday April 18, 2020 @11:43AM (#22908471)
The GIL is not a total ban on thread parallelism. It's a significant obstacle, but not a complete stop.

Besides, Go has its own set of problems with parallelism. None of those are best in class.

Re:Why do we need Flask, Celery, and Redis? (2019) by throwaway894345 on Saturday April 18, 2020 @04:21PM (#22910782)
This comment makes it sound like Go and Python are pretty much in the same class because neither is perfect. I invite anyone who thinks this way to write considerable (shared-memory) parallel code in both languages and see which they prefer.

As for Go's "set of problems with parallelism", they're pretty much just that sharing memory is hard to do correctly without giving up performance. No languages do this well; Rust and Haskell make it appear easier by making single-threaded code more difficult to write--requiring you to adhere to invariants like functional purity or borrowing. If you're writing Python, you very likely have values that are incompatible with these invariants (you want to onboard new developers quickly and you want your developers to write code quickly and you're willing to trade off on correctness to do so).

Go is absolutely best-in-class if you have typical Python values.

Re:Why do we need Flask, Celery, and Redis? (2019) by dnautics on Saturday April 18, 2020 @10:11PM (#22912886)
Why bother writing shared-memory parallel code if it makes your life so hard? Most of the time you're i/o bound, or network bound, or storage bound. Being compute bound is exceptionally rare these days.
Re:Why do we need Flask, Celery, and Redis? (2019) by throwaway894345 on Sunday April 19, 2020 @12:08PM (#22916753)
I do it when I have to do it. Most things are I/O bound, but sometimes things are compute bound. And since Python is about two orders of magnitude slower than Go (and Python also makes it much harder to optimize straight-line execution because it lacks semantics for expressing memory layout), you tend to need parallelism more often than you would in a fast language. Sometimes you can leverage pandas or write a small chunk in C, but very often those options aren’t available, and naively throwing c at the problem can make your performance worse.
Re:Why do we need Flask, Celery, and Redis? (2019) by marcosdumay on Saturday April 18, 2020 @06:28PM (#22911649)
> Go is absolutely best-in-class if you have typical Python values.

"Best in class" is not a relative term. Neither Go nor Python are appropriate choices for highly parallel intercomunicating code. Yes, Python is more limited than Go here, but hardly makes a difference when you avoid it.

Haskell and Rust do make it easier, by forcing developers to organize their code in a completely different way. Erlang does the same, with a different kind of organization. None of those languages are more difficult to program in, but yes, they are hard to learn.

Re:Why do we need Flask, Celery, and Redis? (2019) by jatone on Sunday April 19, 2020 @09:34AM (#22915768)
> Neither Go nor Python are appropriate choices for highly parallel intercomunicating code.

thats an extraordinary claim which needs evidence.

Re:Why do we need Flask, Celery, and Redis? (2019) by marcosdumay on Sunday April 19, 2020 @10:36AM (#22916172)
Extraordinary?

Name any single Go feature aimed at helping parallel computing.