News for hackers. Stuff that matters.
Re:Go, Python, Rust, and production AI applications by agent281 on Tuesday March 12, 2024 @11:06AM (#39680406)
My unpopular opinion is that pandas specifically is terrible for data engineering. The type inference when loading data from files can result in fragile jobs. In my experience, it's also surprisingly slow for a lot of data engineering tasks. IMO, it should just be used for analysis.

Python's parallelism story also holds it back, but that may be improving soon. It's unlikely it will be as nice as Go's though.

Re:Go, Python, Rust, and production AI applications by isoprophlex on Tuesday March 12, 2024 @12:08PM (#39681186)
Is that unpopular? I rarely meet people who actually enjoy wading through the baroque, underperforming forest of footguns that is the pandas API.
Re:Go, Python, Rust, and production AI applications by agent281 on Tuesday March 12, 2024 @12:58PM (#39681892)
When I would interview data eng candidates, a large subset would import pandas before reading the problem statement. When I've complained about it, a lot of people are surprised to find a data engineer who doesn't like it. That said, it might just be the companies I've worked at. ¯ \ _ ( ツ ) _ / ¯
Re:Go, Python, Rust, and production AI applications by isoprophlex on Tuesday March 12, 2024 @02:44PM (#39683258)
Keep on educating them I'd say ;)