News for hackers. Stuff that matters.
Re:Pioneer manipulates scores to pick winners by danicgross on Friday July 26, 2019 @01:21PM (#20536418)
Pioneer founder here. As noted by almost all commenters in this thread, we do rely on expert voting. It's in our FAQ. We take pride in that! I believe mixture of moderation _and_ quantitative metrics are the correct path towards accomplishing our goal.

We've all seen what happens when you let the algorithms run free.

We've got much to fix in terms of making the game more _fun_. Point systems are best when you _understand_ the underlying mechanism. There's a tension between that and moderation. Losing should feel educational, not frustrating. We've got some ideas on how to fix this, and we're burning the midnight oil implementing them.

(BTW, we're very open to any product suggestions here! In case you're wondering what the ruckus is about: the Pioneer idea is to make an online game that rewards productive, creative behavior. Which beckons the question: how do you quantify productivity in different domains?)

Re:Pioneer manipulates scores to pick winners by loresome on Friday July 26, 2019 @01:46PM (#20536667)
When one component of the system has infinite power over the other (moderation over quantitative metrics here), you can't call that a mixture IMO.

I would suggest these three things: (1) separate the concept of "experts" and "organizers/developers". When I hear "experts" I assume "independent experts". (2) change deceiving language on the landing page. Quoting: "All you need to do is convince other participants that your project is worth doing." and "Every week other participants will give you feedback and points. The more progress you make, the higher your score will be." No mention of experts, yet alone organizers at all. (3) Make the points transparent. That includes breaking down voting vs experts vs whatever and disclosing the details of the scoring and matchmaking systems. Doesn't have to be exact spec/formula obviously, but general details.

Re:Pioneer manipulates scores to pick winners by danielgross2 on Friday July 26, 2019 @01:57PM (#20536802)
Thanks OP! What I find curious is why we couldn't continue our existing exchange over email.

(noprocrast correctly suggested I get back to work, so made a new account.)

Re:Pioneer manipulates scores to pick winners by loresome on Friday July 26, 2019 @02:01PM (#20536842)
Well, I wrote to you (Pioneer team) multiple times over this week. I actually sent all the materials for this article yesterday.

Edit: re-read your comment more carefully. To answer: because the exchange was not productive. I only saw general comments equating to "you don't understand our opaque system" without even acknowledging the arguments I was making.

To add to my previous comment. What separates "expert voting" from "leaderboard manipulation" the most in my opinion is the access to the leaderboard state. If experts vote in isolation, based solely on the project info and reports it's a "mixture of signals". If they have unlimited power, and change the scores to shape the leaderboard to a desirable state, it's manipulation.

Re:Pioneer manipulates scores to pick winners by dnautics on Friday July 26, 2019 @04:06PM (#20537949)
I agree that pioneer is super sketchy about it's exper scoring system. I think in practice it's more like peers float your app to the top of the applicant pool and then experts don't bother themselves with the ones ranked lower, so it's not as off the peers are useless. At least thats what I figured when I applied last year, and gave up when I realized that the experts were not really selecting "outsiders"
Re:Pioneer manipulates scores to pick winners by mr_puzzled on Friday July 26, 2019 @02:10PM (#20536939)
Protip : never try to hide your shortcomings. Embrace criticsm and scrutiny. If you do it right, these instances should only help you become a better team and build better products. How you respond to scrutiny pretty much gives an idea of how you handle yourself when the going gets tough, and comments such as these just don't inspire confidence.
Re:Pioneer manipulates scores to pick winners by hjk05 on Friday July 26, 2019 @03:04PM (#20537457)
Do experts take into account previous performance and overcompensate current score to ensure previous top players are pushed out the top 10?

Data seems to suggest this, which is going to extreme lengths beyond “expert votes”. And is more akin to changing the rules at the last minute to avoid living up to the promises put out by the platform.

If this is not the case then perhaps you can release more data to illuminate the oddities highlighted by the article?

Re:Pioneer manipulates scores to pick winners by opencl on Friday July 26, 2019 @01:36PM (#20536560)
You should probably put the FAQ link somewhere other than the very bottom of the page, it's quite easy to miss as-is.