> is the onus on an artist or on an "auction house" to vet buyers. If post sale it turns out the money was fraudulent, does the artist need to pay it back?

No. Normally you have to return items that were stolen from someone even if you purchased them without knowing they were stolen. But money is an exception. See:

https://en.wikipedia.org/wiki/Nemo_dat_quod_non_habet

(I don’t know whether Bitcoin would be treated as money for these purposes…)