News for hackers. Stuff that matters.
Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by sseagull on Saturday April 08, 2023 @09:10AM (#35492781)
Maybe I'm just a little skeptical today, but I'm not super enthusiastic about this.

1.) Adding another language significantly increases complexity in general. Now I see several packages (pydantic, pydantic-core, pydantic-settings). Issues can crop up in either language, or in the translation.

2.) Your typical users are python developers, many which don't know rust, and therefore would be unable/unwilling to help contribute or help debug issues. Debugging an issue with pure python is easier (ctrl-click in my IDE will take me to the code and I can look and see what is going on, add print statements, etc, even in external libraries).

3.) At least in my use cases (and we use pydantic quite a bit), performance increases there will be negligible compared to things like data transfer (network, db) and the rest of my code that's written in python.

Anyway, I do love pydantic, and hope this is just a drop-in replacement and I don't have too many issues.

Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by crabbone on Saturday April 08, 2023 @09:29AM (#35492888)
For a while, I was meaning to write a package manager for Python, out of total frustration with existing ones.

If I ever find the time and will to actually commit to such a project, I will never use Python for it. There's an obvious reason, which in my case can work as an excuse: not wanting circular dependencies (both pip and conda today depend on Python version and a bunch of Python packages, which makes them flaky / needing multiple versions depending on what Python version is used). But that's just part of the problem.

Python community today houses the worst programmers I know to exist. The quality of community-produced code was and is still degrading fast. I think, Python overtook PHP on this metric a while ago, we just didn't notice it.

So, if I ever make another publicly distributed library for Python, I'm not writing it in Python. Both because I don't want any input from people writing in Python (exclusively) and because with each new version of the language it gets worse. Each new Python release is a step forward and ten steps back: they do fix some bugs, and every now and then clean things up, but they add ten times more garbage to the language than the work they put to make it genuinely better. I feel like if I'm going to chase Python language iterations, I'll be just plugging holes on a sinking ship, and eventually will just run out of stuff to plug the holes with.

Finally, Python doesn't have its own strategy for making Python code to go fast. What I'm talking about is stuff like const correctness in C, or no dynamic memory allocation in C, or eliminating dynamic dispatch in C++ or Java and so on. Many languages have their own "min-game" developers play to make programs faster. In Python, the name of this mini-game is "write in C". It doesn't have to be C specifically, but it's quite obvious that no tricks and backflips you can do in Python alone will ever be comparable performance-wise to simply rewriting even not-very-efficient code to C.

Python is a delivery mechanism today. It's a good delivery mechanism because many people use it. But it's an awful platform for programming. So, if you want comfort in programming and have many users: write in C, and add bindings in Python.

Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by ilyt on Saturday April 08, 2023 @09:47AM (#35493058)
> Python community today houses the worst programmers I know to exist. The quality of community-produced code was and is still degrading fast. I think, Python overtook PHP on this metric a while ago, we just didn't notice it.

JS topped that metric long time ago;

But it is a curse of every language that gets popular as new developer's first language

Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by devonkim on Saturday April 08, 2023 @09:58AM (#35493172)
A long time ago I remember people saying that JavaScript is the most popular language that programmers don’t learn. A language with a sufficiently low bar for mass adoption by non-programmers in theory should be useful without needing to know the ins and outs. Yet, I would argue still that the world’s most popular programming language isn’t even either JS or Python or PHP - it’s Excel. When we constrain a programming operating environment and language elegantly it’s quite impressive what is accomplished. We as programmers tend to forget oftentimes that computers and all that stuff is just a means to an end.
Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by crabbone on Sunday April 09, 2023 @05:32AM (#35501148)
Can you write extensions / modules for VBA used in Excel? -- It's an honest question. I simply haven't used any Microsoft product in a very long time... If not, it's probably not a good delivery mechanism, even if very popular.
Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by devonkim on Sunday April 09, 2023 @09:51AM (#35502725)
There’s various macro extensions and plugins available including .NET but these options tend to have a long history of horrible security problems because they are general purpose languages. While possible to crash spreadsheets with simple formulas they’re not going to steal your local credentials off disk probably.
Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by MrJohz on Saturday April 08, 2023 @03:43PM (#35496873)
I'm not entirely sure that's the case. I agree, Javascript is very popular as a first language and an introduction to the world of programming, especially with all of the frontend bootcamp-type courses. But in most of these courses embed a lot of healthy "best practices". Sure, there's often a sense of cargo cult appeal rather than a full explanation, but, for example, most new developers seem to at least have a basic knowledge of git and version control (even if it's often treated as a set of mystical incantations that must be performed before programming). The tooling also helps a lot here, for example by installing dependencies locally and saving a lock file to prevent automagical updates.

Meanwhile, a lot of people who write Python are not necessarily developers, they are scientists or data analysis who just want to turn their data into conclusions. So things like version control are kept largely to one side as unnecessary complexity, and dependency management is often global, ad-hoc, and prone to breaking. Even things like Jupyter - which by itself can be a great tool - tend to push less experienced developers towards spaghetti code and difficult to reproduce results.

In fairness, I think it's difficult to compare these situations and make hard conclusions about which one is better. Ultimately, in both ecosystems, there are some fantastic developers producing excellent tools, frameworks, and libraries. The problem, to the extent that it exists, lies in the low barrier to entry, and the accessibility of these languages, and that's something that's hard to complain too hard about!

Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by crabbone on Sunday April 09, 2023 @05:29AM (#35501135)
I remember JavaScript from the time it wasn't considered a language anyone would write by hand (RAD controls? Something similar generated by MSVS?). It was supposed to be generated because most people involved with it looked at it with disgust. This tradition, sort of, continues to this day with the most prominent member of the family being TypeScript. So, I'd say, JavaScript community is more aware of the awfulness of their language, and have created an ecosystem where they constantly make efforts to not to write in JavaScript (of course there's also the opposite camp of people who really want to write in JavaScript).
Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by physPop on Saturday April 08, 2023 @02:23PM (#35496133)
no one should be writing C unless they have a very specific use case. There are so many better safer options like Rust or compile-to-c like Zig, Odin, and Nim.
Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by ledauphin on Saturday April 08, 2023 @09:14AM (#35492805)
this seems like a reasonable objection.

for my part, i wish somebody would write a data modeling library for Python where the underlying objects are themselves Rust objects, such that they could be handed off to Rust code with no translation penalty, but where the data model definition happens completely on the Python side.

This would be helpful on larger teams where everyone needs to be able to contribute to the data model but not everyone is comfortable writing Rust.

Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by crabbone on Saturday April 08, 2023 @09:38AM (#35492963)
All Python objects are C structs already... that's in part why binding between Rust and Python is possible.

Rust doesn't really have objects though. It has structs. But these are the definitions that exist ad definition time. They disappear into the ether at runtime, so, I'm not sure how such communication between Python and Rust could've happened. Or, in other words: did you mean communication at run time or at compile time? Does Python, in your version of events, generate Rust source code, or does it talk to a program written in Rust?

Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by orf on Saturday April 08, 2023 @09:18AM (#35492828)
It's not possible to do with no translation penalty. Rust (and other languages like it) need the sizes of objects known at compile time in order to allocate space on the stack (and within struct fields).

One way you could do it would be to add dynamically dispatched ".get_attribute(str) -> Option<...>" methods which adds a lot of overhead.

Another way would be to have some bastardised `build.rs` script that constructs Rust structs from Python class definitions. This keeps the definitions in Python but obviously doesn't track dynamic changes to it at runtime.

Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by jeroenhd on Saturday April 08, 2023 @11:22AM (#35494116)
I think your objections are fair, but I also think it's not that much of a difference. Most common Python projects seem to have a native component somewhere (often written in C) because pure Python has rather bad performance.

Separating the packages shouldn't be too much of a problem because dependency management should keep the versions in sync. Broken releases can occur, but that also happens in single package models.

I think the performance benefit is significant enough to excuse the barrier of entry for maintainers. Large projects will be unlikely to see much performance improvements because there are so many other slowdowns in other layers of code and dependencies, but every projects that fixes their performance reduces the impact.

I think Python shines as a language when it's used to glue together native code. Writing correct C and Rust is hard and most applications are usually just applying some custom logic to existing code. In my opinion, the more packages are available in native, compiled packages, the better; it doesn't matter if the native code was written in Rust or hand-crafted assembly, as long as the Python calls work without having to restructure the entire project.

It looks like this rewrite is a drop-in replacement with some minor renames and API changes. It's still a major version upgrade so there may be more differences that I can't see, but the migration guide (https://docs.pydantic.dev/blog/pydantic-v2-alpha/#migration-...) seems simple enough. There's nothing in there that I wouldn't expect if they hadn't rewritten the entire thing in a different language anyway.

Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by manfre on Saturday April 08, 2023 @11:23AM (#35494139)
2) from my experience maintaining open source packages, many would report issues with little to no troubleshooting. A very small subset would look at the project code. The pedantic v1 code is not the easiest to digest. Changing v2 to rust will likely not have much of a change with regards to user reports.
Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by shalabhc on Saturday April 08, 2023 @05:50PM (#35497992)
I'd be curious is Cython was evaluated as an alternative. With less of an impedance mismatch with Python and capable of similar speedups it might be a good fit for this use case.
Re:Pydantic V2 rewritten in Rust is 5-50x faster than Pydant... by bitwize on Sunday April 09, 2023 @11:46AM (#35503812)
> Adding another language significantly increases complexity in general.

True, but no language -- not even Lisp -- sparks joy the way Rust does. Developers are more than willing to endure the additional complexity for the benefits Rust provides: fearless performance or low-level capability. Unlike C or C++, safe Rust is within grasp of the average Python developer.