Forget email, say you have a app that scans links in comments for maliciousness. You rely on an internal api for checking against a known blacklist, which follows shortened links first, and an external api from a third party. You want the comment to appear to submit instantly to the poster but are comfortable waiting for it to appear for everyone else. What are your options?
You could certainly use message queues and workers. If you’re cloud native maybe you leverage lambdas. Maybe you spin up an independent service that does the processing and inserting into the database in the background, and all you need to do is send a simple HTTP request on an internal network.
Your solution depends on your throughout requirements, the size of your team and their engineering capabilities, what existing solutions you have in place. Everything has its pros and cons. Pretending that celery/redis is useless and would be solved if everyone just used Java ignores the fact that celery and redis are widely popular and drive many successful applications and use cases.
Right now, we do hypercorn multiproc -> per-proc quart/asyncio/aiohttp IP-pinned event loop -> Apache arrow in-app cache -> on-gpu rapids.ai cache.
But not happy w event loop due to pandas/rapids blocking if heavy datasets by concurrent users. (Taking us back to celery, redis, etc, which we don't want due to extra data movement..) Maybe we can get immutable arrow buffs shared across python proc threads..
Ideas welcome!
More off-topic (or, rather, on-topic), I find lambdas great for things like a static website that needs a few functions. I especially like how Netlify uses them, they seem to fit that purpose exactly.
Me too! It makes me irrationally angry when people regurgitate linguistic clichés. I was already mad with:
"python does not have a go-like concurrency story"
when it would be enough (and 1000x less cringe) to say:
"python does not have go-like concurrency"
I think these mindless clichés make language really ugly and dysfunctional, and even worse they are thought-stoppers, because they make the reader/listener feel like something smart is being said, because they recognize the "in-group" lingo. In my experience, people get really offended when you point this out. It's kind of an HN taboo to discuss this. Which is also interesting in itself.
Going forward we should pay more attention to our communication use cases. Btw: I wonder if we can stack several of these clichés. For example: "leverage" + "use case" = "leverage case".
I've found it's a taboo to discuss anything even slightly personal. People are averse to feeling bad, so criticism needs to be extremely subtle in order to not offend.
> Btw: I wonder if we can stack several of these clichés. For example: "leverage" + "use case" = "leverage case".
I hate you for even thinking of this.
The personal association you made between "discussing anything even slightly personal" and "criticism needs to be extremely subtled" makes it sound that your problem isn't language or Orwellian discourse but the way you subconsciously link discussing personal matters with harshly criticising those you speak with for no good reason.
If your personal conversations boil down to appease your own personal need to criticise others then I'm sorry to break it to you but your problem isn't language.
You also went to "Orwellian discourse", which has a specific meaning, from a text by Orwell I mentioned. It seems to me like you got personally offended, interpreted my comment in the most uncharitable way, and chose to lash out at me instead, and I'm not sure why. I wasn't even talking about anyone specifically.
I repeat, your problem is not language. Your problem is that you manifest a need to criticize others. That problem is all on you.
https://news.ycombinator.com/item?id=22911497
PS: I work with like 65-70% of that stack daily
This shows from the fact that you're saying "criticize" like it's a bad thing.
If you don't need all that, then it's not a problem in Python either. You don't even need asyncio, you can just use a ThreadPool or a ProcessPool, dump stuff with pickle/shelve/sqlite3 and be on your way.
It's not in the standard library, and there are probably other options too now (not a heavy Python user any more), but Python has had easy parallelism for at least a decade (probably longer).
And it's showing great promise ...while it could be delayed.
https://www.mail-archive.com/[email protected]/msg108063...
2021 is probable going to be the "Gone Gil" moment !
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
This runs f(1), f(2), and f(3) in parallel, using a pool of five processes.Python standard library since 2.6. Pretty much the definition of "in python".
On its own, yes. For webapps, you can easily combine it with a multi-process WSGI server (like gunicorn or similar).
Besides, Go has its own set of problems with parallelism. None of those are best in class.
As for Go's "set of problems with parallelism", they're pretty much just that sharing memory is hard to do correctly without giving up performance. No languages do this well; Rust and Haskell make it appear easier by making single-threaded code more difficult to write--requiring you to adhere to invariants like functional purity or borrowing. If you're writing Python, you very likely have values that are incompatible with these invariants (you want to onboard new developers quickly and you want your developers to write code quickly and you're willing to trade off on correctness to do so).
Go is absolutely best-in-class if you have typical Python values.
"Best in class" is not a relative term. Neither Go nor Python are appropriate choices for highly parallel intercomunicating code. Yes, Python is more limited than Go here, but hardly makes a difference when you avoid it.
Haskell and Rust do make it easier, by forcing developers to organize their code in a completely different way. Erlang does the same, with a different kind of organization. None of those languages are more difficult to program in, but yes, they are hard to learn.
thats an extraordinary claim which needs evidence.
Name any single Go feature aimed at helping parallel computing.
Performance is pretty close for both. I was disappointed to not see Python get an official green thread implementation. The counter-argument I commonly see cited is https://glyph.twistedmatrix.com/2014/02/unyielding.html. I personally don't find it to be a very convincing argument.
The coroutine and queue model is the same right ?
Cool thing in 3.8:
Running python -m asyncio launches a natively async REPL.
The vast majority of the time, gevent's monkeypatching works without any issues. With asyncio, you basically have to rewrite everything from the ground up to always use the new async APIs, and you can't interact with libraries that do sync I/O.
Can you provide some context for this statement? I've used Python asyncio extensively in Fargate (no ASGI frontend), and the developer experience is far from Go; however, I don't see how an ASGI framework can fix this. It seems like it offers the same general course-grained parallelism that you get from a containerized environment like Fargate except that Fargate abstracts over a cluster while ASGI frameworks presumably just abstract over a handful of CPUs.
For example, we have a large data structure that we have to load and process for each request. We want to parallelize the processing of that structure, but the costs to pickle it for a multiprocessing approach are much too large. We've considered the memory mapped file approach, but it has its own issues. We're also looking at stateful-server solutions, like Dask, but now we're talking about running infrastructure. In Go, we could just fork a few goroutines and be on our way.
i dont claim to have expertise in your business domain, but you should get the answer here.
(1) it's undersupported (e.g. you can in theory download s3 files using async botocore, but in practice it is hard to use because of strict botocore version dependencies)
(2) it isn't natural - once you're in the event loop it makes sense, but using an event loop alongside normal python code is confusing at best.
(3) it got introduced too late. The best async primitives are only available in pretty recent versions of python3.
The difference with go is that it go has these primitives built in from the beginning and using them doesn't introduce interoperability problems.
If you have CPU intensive workload, an optimizing compiler can help. Well, for that you have C or other languages with more mature and aggressive compilers.
Of course if you are stuck with Python it's better than nothing.
You can't say that when Python shoves Pip/Poetry/VirtualEnv/Black/Flake and what not into people. In contrast, Go has built-in package management and gofmt. Python is essentially gate-keeping new devs.
Kubernetes - which is one of the biggest projects built in go - has been struggling with dependency and package management.
Here's the CTO of Rancher commenting on his struggles
https://twitter.com/ibuildthecloud/status/118752909888666419...
https://twitter.com/ibuildthecloud/status/118753821015230873...
This is not trivial stuff..and it shouldn't be trivialised into a go vs python flamewar. Because it can't be.
To work with Python packages, you have to pick the right subset of these technologies to work with, and you'll probably have to change course several times because all of them have major hidden pitfalls:
* wheels
* eggs
* pex
* shiv
* setuptools
* sdist
* bdist
* virtualenv
* pipenv
* pyenv
* sys.path
* pyproject.toml
* pip
* pipfile
* poetry
* twine
* anaconda
To work with Go:
* Publish package (including documentation): git tag $VERSION && git push $VERSION
* Add a dependency: add the import declaration in your file and `go build`
* Distribution: Build a static binary for every target platform and send it to whomever. No need to have a special runtime (or version thereof) installed nor any kind of virtual environment nor any particular set of dependencies.
Python packaging is a mess, but Go doesn't even bother. "Just download from some VCS we'll pretend is 100% reliable and compile from source" is not a packaging solution.
You can certainly debate the difference in uptime between specific services; I don't know either way, but if you told my that PyPi had higher uptime than GitHub, I'd believe you... but that's kinda missing the point. If you depend on an online service to host your release artifacts, if and when that service goes down, it's gonna hurt.
Meanwhile, Python's packaging wars continue to rage on. Go's is simple: a release is a tag in a VCS repository. I'm sure there are issues with that as well, but that should come as no surprise, considering there are issues with literally every packaging solution. At any rate, there's little moral difference between downloading a tarball (or a wheel, or... whatever), vs. pulling a tag from a git repo. It requires equal levels of trust to believe that no one has tampered with prior releases in both cases.
I'd like to also point out that I don't have a dog in this race. I've done a little Go here and there, but frankly I don't like the ergonomics of the language too much, so I stay away from it. I've done (and continue to do) a decent amount of Python. I like the language, but tend to prefer strongly-typed, functional languages, and languages with performant runtimes, so I tend to only use it for smaller projects.
go has support for a proxy system, tooling is still immature though.
that's what vendoring is for and the proxy cache. this problem hasn't existed since like go 1.8 and is completely resolved in go1.14.
If nothing else, Go lets you distribute a static binary with everything built in, including the runtime. Python's closest analog is PEX files, but these don't include the runtime and often require you to have the right `.so` files installed on your system, and they also don't work with libraries that assume they are unpacked to the system packages directory or similar. In general, it also takes much longer to build a PEX file than to compile a Go project. Unfortunately, PEX files aren't even very common in the Python ecosystem.
> Pip/Poetry/VirtualEnv
"packaging" refers to the way the language manages dependencies during the build and import process, not how you distribute programs you have built.
Python has a deservedly poor reputation here, having churned through dozen major overlapping different-but-not-really tools in my decade and a half using it. And even the most recent one is only about a year into wide adoption, so I wouldn't count on this being over.
Go tried to ignore modules entirely, using the incredibly idiosyncratic GOPATH approach, got (I think) four major competing implementations within half as long, finally started converging, then Google blew a huge amount of political capital countermanding the community's decision. My experience with Go modules has been mostly positive, but there's no really major new idea in it that needed a decade to stew nor the amount of emotional energy. (MVS is nice but an incremental improvement over lockfiles, especially as go.sum ends up morally a lockfile anyway.)
With elixir, you run "mix release" and the release pipeline is set up to automatically gzip the release (it's one line of code to include that). Shoot the gzip over (actually I upload to s3 and redownload), unzip, and the entire environment, the dependencies, the vm, literally everything comes over. The only thing I have to do is sudo setcap cap_net_bind=+ep on the vm binary inside the distribution because linux is weird and, as they say, "it just works".
But if Python finally "picks" poetry, sticks with it for a few years and incrementally fixes problems rather than rolling out yet another new tool, that will also be better.
You can only identify the end of the churn for either retroactively. Python just looks worse right now because it's been around longer.
go: tends to wait and implement something once the problem is understood. took 2 years after go maintainers decided to solve the dependency issues. and as of the latest release its finally been labelled production ready.
and honestly the proxy issues were not real. go modules was still optional. you could just turn it off.
python is how old now? couple decades? and it has only gotten worse over time.
> go: tends to wait and implement something once the problem is understood.
Go's modules provide no additional "understanding" over any of the other Bundler-derived solutions in the world. MVS was the primary innovation, but wanting checksum validation means I have to track all the same data anyway.
> took 2 years after go maintainers decided to solve the dependency issues
This is revisionist history. There were other official "solutions" before ("you don't need it", "vgo is good enough", and "we'll follow community decisions"). If this one sticks, it's fine. But you can't say it's good now just because it's the one we have now - it's good now if it's the one we still manage to have in five years.
Go's track record is not "good" (in that regard I think only Cargo qualifies). At best it's "mercifully short."
> and honestly the proxy issues were not real.
Documentation was poor, the needed flags changed shortly before release, the design risks information leaks, and the entire system should not have been on by default for at least one more minor version.
> python is how old now? couple decades? and it has only gotten worse over time.
Yeah, that's exactly why I said "Python just looks worse right now because it's been around longer." It hasn't gotten worse though, it just also hasn't stopped churning. And if Go doesn't stop churning, in 10 years it will look the same.
The age argument works both ways - multiple major versions of Python predate Bundler. Go has no excuse for taking so long to reinvent "Bundler with incidentals", just like every other language.
There was also a bad decision of using Python code for installation (setup.py) instead of a declarative language.
Most of that issues are actually fixed in setuptools if you put all settings in setup.cfg and just call empty setup() in setup.py.
Like here: https://github.com/takeda/example_python_project/blob/master...
Fyi - python ASGI frameworks like fastapi/Starlette are the same developer experience as go. They also compete on techempower benchmarks. Also used in production by Uber, Microsoft,etc.
A queue based system is used for a very different tradeoff of persistence vs concurrency. It's similar to saying that the usecase for Kafka doesn't exist because go can do concurrency.
Running "python -m asyncio" launches a natively async REPL. https://www.integralist.co.uk/posts/python-asyncio/#running-...
Go play with it ;)