On occasion, I've fired up pandas just to sanitize a CSV file and drop malformed rows as preparation to bulk ingesting into a database:

  import pandas as pd
  pd.read_csv('bad_file.csv', error_bad_lines=False).to_csv('good_file.csv')

It's not efficient (reads everything into memory), but read_csv is robust when it comes to handling embedded unescaped quotes/commas/etc., and supports dropping rows with the incorrect number of columns due to anomalies it can't handle.

Genuine question - would you be willing to spend money for a better version of pandas?

Better in some generic sense of lighter, faster, better API.

I share your implied concern that pandas can be quite large and I personally disagree with a lot of the design decisions when it comes to the pandas API, but building an alternative tool would be a full time job. Unfortunately, there is no mechanism to support Python library developers and the expectation is for Python libraries to be free.

I'm curious how many people would be ok paying for a Python library.

I think that would be an uphill battle and very hard to succeed financially. I agree with you regarding the API being a mess, but pandas is so heavily entrenched in the datascience space (in Python land) that it is almost impossible for a free replacement to take over, let a lone a paid library.

And pandas is valuable particularly because it has so many users. I'd worry that a paid product would stagnate over time, or change to meet the needs of its largest customers, leaving me behind.

I go out of my way to support open source projects. Closed source would be a much harder sell.

I have so many things to say about this but I also want to remind you that Wes is working on pandas 2.

https://dev.pandas.io/pandas2/goals.html

Good to know!

I thought that Wes had said a while ago that he was taking a break from working on Pandas

If you need to scale out or speedup pandas, there's Modin https://modin.readthedocs.io/en/latest/ (which uses Ray from)