Re:Amazon’s Mechanical Turk Has Reinvented Research
by mrgordon on Friday May 18, 2018 @06:18AM
(#17100218)
CrowdFlower / Figure Eight works with a lot of annotators who have signed NDAs and work in secure locations where they can't duplicate materials etc.
Re:Amazon’s Mechanical Turk Has Reinvented Research
by ayw on Friday May 18, 2018 @10:50AM
(#17101900)
Alex from Scale (www.scaleapi.com) here! We've taken an extremely quality-first approach and build out large workforces for datasets with high quality requirements and complexity. For example, we do a bunch of LIDAR / 3D labeling (https://www.scaleapi.com/sensor-fusion-annotation) which is very complex and labor intensive, and provide extremely high quality that would not be possible otherwise.
Re:Amazon’s Mechanical Turk Has Reinvented Research
by deepakputhraya on Sunday May 20, 2018 @02:51AM
(#17111581)
Deepak from Playment(https://playment.io/). We help companies label data. We have dedicated managers for projects who take care of training annotators and ensuring quality output. We have a large number of annotators who are pre-trained on different annotations, and we train them for different specific business use-cases. Reach out to us if you are looking for fully managed quality annotated data.
Links:
https://playment.io/image-annotation/
Re:Amazon’s Mechanical Turk Has Reinvented Research
by cwheatley on Friday May 18, 2018 @02:04PM
(#17103570)
I work directly with several top-tier research universities use us for public opinion polling mostly, as we offer a representative sample at a great price point.
Lucid is the creator of the world's largest programmatic survey sample marketplace. We have API integrations with hundreds of US and international panels that allow us to target very specific audiences across dozens of sources, ensuring excellent feasibility.
Happy to discuss any needs the YC community has.
- Cullen Wheatley
Articles for more info: https://techcrunch.com/2017/04/10/lucid-60-million/
https://greenbookblog.org/2017/05/16/ceo-series-how-lucid-ce...
Re:Amazon’s Mechanical Turk Has Reinvented Research
by mwexler on Friday May 18, 2018 @09:41AM
(#17101345)
Seconded. Quality control is a nightmare with Turk, often requiring stimuli to be labeled multiple times and have a variety of judging approaches to "crown a winner". Companies like DefinedCrowd (https://www.definedcrowd.com/) have taken a quality-first approach which gives much better data, but of course, at a cost.
Re:Amazon’s Mechanical Turk Has Reinvented Research
by therealmarv on Friday May 18, 2018 @05:58AM
(#17100121)
try PYBOSSA, open source crowdsourcing platform, you host it yourself, you are in control of your data.
Re:Amazon’s Mechanical Turk Has Reinvented Research
by k__ on Friday May 18, 2018 @06:08AM
(#17100170)
I think the problem is more that you give the data to the turkers than to give it to Amazon.
Re:Amazon’s Mechanical Turk Has Reinvented Research
by codydh on Friday May 18, 2018 @08:27PM
(#17105935)
Moving to in-house annotators is probably the smart strategy. However, for tasks that are easily done by naive annotators, Prolific.ac might be a great source. I've had good luck getting quality data there, and they enforce a minimum hourly wage that, while not truly livable, is still heading in the right direction.
- business-related data too sensitive to share with strangers (contractual obligations, too much risk)
- some tasks required non-trivial subject matter expertise and context to annotate properly (quality control issues)
For this reason, we gradually moved to an in-house team of long-term annotators. It's not much more expensive (moms on maternity leave, students…), but infinitely more flexible and safer for our purposes. YMMV.