Given that a business pivoted away from exactly this, when they had this up and running, I'd advise putting in some effort to find out why they did so first.

Sounds sweet. I bet a lot of companies relying on mturk built this for themselves and then sell a higher value service with better margins. You could build something right in the middle.

I know Stanford's research teams all use a common interface to mturk that keeps profiles of turkers on their side so they know who to solicit for upcoming surveys, conduct longitudinal studies, etc. I've always wondered why more universities didn't follow suit.

I built a side project called cogmint based on the insight that simple scoring and ranking of workers was valuable. I ended up building my own worker interface instead of using mturk because it wasn't much additional effort on top of the scoring logic I was building anyway. Perhaps other serious companies came to the same conclusion I did with my hobby project.

As AMT’s largest requester, we have longitudinal (~3yr) worker quality data for over 85% of respondents. Feel free to DM for details

How do you deal with respondents who just created their account?

Technical and Demographic user metrics are federated across the platform (we supply tasks to other labor markets online, essentially where bots and farms typically attack) so we leverage priors and shared similarity weights. Behavioral user metrics aren’t the only way to evaluate a respondent

Sounds a little meta! Managed managed labeling workflow. I do wish Amazon would do a better job with this, and I also have no doubt if a company like this sprang into existence and was successful, Amazon would clamp down on it in some way — whether by changing terms to disallow such a business, cutting off MT access to the business, or just copying it.