Correct, that's what I mean by a decently-sized QA department. I would never base the score directly on end-users opinions, although the opinions of end-users could certainly be used to initiate the verification process:

Customers say something is fake --> raises ticket to QA department --> a few automatic orders are placed to different "fake" addresses over a period of weeks and months --> these delivered items are authenticated --> Reputation score modified --> customers are followed up with by the call center agents.