Re:GPTs Are GPTs: An Early Look at the Labor Market Impact P...
by ebauch on Saturday March 25, 2023 @04:37PM
(#35307055)
It is stated that they use the same annotators that trained/filtered chatGPT’s output. I would assume its a rather large group (my company has 10 auditors in Nicaragua). The label biases are mostly stemming from that group and - as suggested - could be removed by using experts in each field to annotate the labels. But given some responses here by experts, I am sure those expert labels would have their very own biases :p
The article seems to describing the labeling here:
> Human Ratings: We obtained human annotations by applying the rubric to each ONET Detailed Worker Activity (DWA) and a subset of all ONET tasks and then aggregated those DWA and task scores at the task and occupation levels. To ensure the quality of these annotations, the authors personally labeled a large sample of tasks and DWAs and enlisted experienced human annotators who have extensively reviewed GPT outputs as part of OpenAI’s alignment work (Ouyang et al., 2022).
I understand the authors, four, did the initial labeling and then asked an undefined set of people to the rest of the labeling.