The current go to solution for the kinds of problems that TabPFN is solving would be something like XGBoost. In general it's a good baseline, but the challenge is always that you need to spend a lot of time feature engineering and tweaking the data representation before something like XGBoost can deliver good performance on your regression or classification problems.

For me the promise of foundation models for tabular data is that there are enough generalizable patterns, so that you need less manual feature engineering and data cleaning.

And kudos to the team, I think it's a really creative application of neural networks. I was always frustrated with neural networks, since they were hard to tune on "structured" data and always under-performed (for me), but we also never had real foundational models for structured data.