Copying my comment from the previous submission[x]:
This was a quite common criticism over r/Python[0] as well, saying that using C modules and then benchmarking "Python" with C isn't fair.
Perhaps it was my wording that caused this confusion, for which I am sorry, but I never meant to compare "pure" Python with C. The point I am trying to make is that, Python with C extensions can be as considerably[1] performant as C code for network or memory bound tasks.
[0]: https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redi...
[1]: It's of course controversial whether 60% throughput is considerable or not.
You don't make that point. You instead make the point that calling C code with glue code is slower than C code alone, and the performance hit of using Python as glue code in your case is a 2x slowdown.
The disclaimer section (which contains this information) was added 44 minutes ago. And it's still misleading. For example about hiredis you write:
> Python extension that wraps protocol parsing code in hiredis.
Where is the mention that the parsing code is actually C code?
------
Last but not least you already posted this a year ago on Reddit[1] and Hacker News[2] (edit: someone else did for HN), but only (partially) addressed criticism very recently.
[1] https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redi...
I am not the one who posted here, why so aggressive?
Do not merge that. I once realized that grep is really fast, and replaced our custom solution with bash script, but I still claim that app is written in idiomatic PHP/python/ruby. This project has a lot of problems that it depends on C code is only a symptom. Noting all those bad things in a project Readme is a trap I fall into way to often it never works.
Mostly the latter but also a bit of the former: I could have optimised it further — at the expense of idiomaticity — but I believe gains would be marginal. Other than that, Python is not that fast so you pay some penalties there too.
> pydis is an experiment to disprove some of the falsehoods about performance
and your comment:
> The point I am trying to make is that, Python with C extensions can be as considerably performant as C code for network or memory bound tasks.
So your point is regarding the falsehood belief that python with heavy use of C extensions is (not) much slower (in this case ~60%) than C? I don't think that such a falsehood belief exists and therefore pydis disproves nothing.
The entire purpose is to demonstrate that you can build a reasonable performance application built on high performance libraries and stick with idiomatic Python code.
1) Write Redis in C. Hard for some devs who only know Python, but good performance.
2) Write Redis in Python with C libraries. Doable for a Python dev, performance hit.
3) Write Redis in pure Python. A fun experiment to see the performance limitations of Python when compared to C running on the same machine.
It feels a bit sensationalist to me, and I see this all the time with projects these days. It's kind of disappointing.
However, if the experiment in question is, "using python glue code, can a developer string together a service that replicates the functionality of redis", then that illustrates what would actually happen in the real world.
For example, for a very junior Python developer, writing something in C is basically like asking a C developer to write assembly- possible, but they'd have to learn a lot of different things to do it.
Similarly, when I write Java, I want to be calling Arrays.sort(myarray) instead of writing my own sort (if the focus is on developer convenience and development velocity).
So, it's just a difference in what you're desiring to measure.
"Decent" is a weasel word, which suggests that the performance hit of following this approach is tolerable or acceptable. Sometimes it is, sometimes it isn't. It really depends on where you place the glue code, and where you expect the hot path to be.
In the case of a self-described "Redis clone" which ends up being Redis itself glued back together with Python, I'm not sure you can make that case.
> The entire purpose is to demonstrate that you can build a reasonable performance application built on high performance libraries and stick with idiomatic Python code.
I'm sorry to say but that isn't much of a demonstration. I mean, Python is already widely used as glue code to hold together high performance libraries, and it's already widely known and understood that gluing together code with Python brings in a significant performance hit. And that's ok, because people do it as a tradeoff between performance and some other kind of value-added.
But in this case I fail to see what value-added is being proposed, other than the personal portfolio side of things.
Well, your claim is that the author has missed the point, because I just quoted the authors point and his intended purpose. So I guess we both agree.
You don't think such a falsehood belief exists, therefore you don't think pydis disproves anything.
On some level, it's like complaining that C# is just using C++ modules because the CLR that interprets the C# byte-code is written in C++.
But it misses the point. You wrote this in Python and you were able to leverage the speed of a much more complicate language for your benefit. Without touching the lower language and without worrying about it's memory-management or lib dependencies. This is a win, and it's part of the beauty of programming!
Btw, my compliments to you on writing this! Really cool project/poc.
> In the past year I think I have finally come to understand the ideals of Unix: file descriptors and processes orchestrated with C. It's a beautiful idea.
We just keep moving up the stack, it seems :)
Source: https://gist.github.com/cookrn/4015437
(edit: formatting)
But doesn't the inclusion of hiredis invalidate the entire point? It's 7500 lines of C specifically for working with Redis.
If you write an application in Python that spends the majority of the time in C-based extensions – for example if I were to write a differential equation GUI using numpy and pyqt – then I think it's still valid to say that I wrote the application in Python. In fact even most of the standard library modules are written in C (e.g. zipfile, io, sqlite3) so it's hard to avoid this. You could say that no program is ever really written in Python but I don't think that's a useful point to make.[1]
But if you write a load of C code that's very specific to your application, wrap it in a Python extension, then call it from Python, then I don't think you could argue that you "wrote it in Python". If someone happened to have written that application-specific extension module for you then the situation is not materially different.
([1] A more useful motion is whether your code is vectorised. That is, if you want to do something with a data structure (e.g. mutate each element, find an element, do matrix multiplication), do you have to loop over the elements in Python or do you make a single call into a library that does the looping in C? Either way you're still ultimately going to be calling C code, if nothing else in the Python interpreter itself, but the distinction is obviously still important.)
(There is an exception I didn't mention: if the zip is encrypted then the zipfile module falls back to pure Python code. The difference is very noticeable!)
Between the heavy use of C libraries and correctness issues (https://github.com/boramalper/pydis/issues/12), I don't see how this project proves that Python is faster or safer than other languages.
That being said, the speed with which it was developed can certainly be seen as Python's strong point.
I also saw the title and clicked on it because I thought to myself "no fricken way...in pure python?" It's a touch baity.
a) aren't writing Python
b) invalidates all of the points for writing in Python. i.e speed of implementation, etc.
Isn't this why people like python?
Just glue together some libraries that someone else has written and then you write a viral blog/twitter post about your <insert trendy thing> that you wrote in just 10 lines of code?
Seriously is 't this the main reason people use it? I.e. it has a big library of stuff so it is easy for beginners and/or non-programmers to use?
I'm struggling to think of any other reasons to deliberately pick python to start something new in, although it is at least a bit more same now with types.
Here's a recent write-up of mine where almost all of the time is spent in C extensions but where it was much faster to develop and code in python: https://www.jefftk.com/p/sharding-the-brigade
And this is not an opinion on how companies measure cost or how one language is better than another but maybe trying to explain, and I think underestimate, popularity of Python needs a little bit more insight.
If performance is good enough, where are robustness and features? How large are the effort gap and the quality gap compared to the original Redis?
A programing language can be thought of in the asbtract, as simply a syntax for computation and programming.
But if you go in the concrete, a programming language implementation translates language into machine code for a particular machine.
From that angle, the Cython interpreter/compiler is pretty bad at doing that. This experiment actually shows it. It is 40% slower even if all it does is orchestrates the calls between C code.
This is true of hash map as well. Python is so bad at generating efficient machine code from its own language syntax and semantics, that it would be unreasonably slow to implement hash map in Python. Why? Because Python is really slow.
In order to speed up programs written in Python, you need to write more of your program in some other faster language, such as C.
Now if you reuse a bunch of components written in fast languages, like hash map being implemented in C, well you avoid the need to write the C code yourself. But it doesn't make Python any more efficient.
In practice though, you might be able to not have to implement 90% of your application, because there exist C libraries that do most of what you already need. Thus only 10% of your application will have degraded performance due to using Python. And for that 10%, it might be Python is a great choice, because you like the language and productivity, readability and all it gives you in order to implement that 10%, and can deal with 10% of the code being slower.
But if there doesn't exist existing fast libraries you can leverage, well more and more of your program will be Python, and thus it'll get slower and slower, unless you start using C yourself and implementing your own C code libraries.
Yes, that's true of hash map as well. For example, if for any reason you needed a hash map that behaved differently than the standard one, you would most likely need to write it in C to get any performance out of it. Writing it in Python would be just too slow.
So to recap, Python is a slow language, because there are no known interpreter/compiler for it that generate fast machine code for some real machine. This is not a falsehood, but the current reality. But it turns out most applications can leverage existing C libraries for a large amount of their functionality, thus making it that you can build an app that is reasonably performant, with the only code you need to personally implement being implemented in Python.
https://devblogs.microsoft.com/aspnet/grpc-performance-impro...
However Python isn't never going to get there.
You might also be interested in my other comment: https://news.ycombinator.com/item?id=25101200
Everyone seems to think that writing expressive code is slow, when really it's because Python is slow and C++ is a lumbering behemoth.
Python isn't even a high-level language, think about it - it's higher level than C but the capability it has for abstraction it gives you is really just interpreted C with classes.
It uses a C Redis client parser (hiredis), uvloop (Python wrapper for libuv) and the Python hashmap implementation.
All the performance critical code is implemented in C.
I appreciate the point that higher level languages can be more than fast enough for many use cases. A tremendous amount of software is written in Python, PHP, Ruby et al after all.
But this is not a good way to make that argument, since it almost refutes it's own point: if performance matters, you apparently implement it in C/C++/Rust.