Thoughts on TurkPrime
If I had to choose between TurkPrime and MTurk, I would favor TurkPrime. This is not a sponsored post!
My views, so far, of TurkPrime are based on two studies of >200 participants (I had to budget), and as a collaborator on a multi-study project, with >500 participants per study. I’ve done about 5-6 other projects on MTurk. I’ve come across blog posts centering mostly on MTurk in general, so I thought some information for those who are considering TurkPrime might be helpful.
Participants appear to pass more attention checks at a pretty high rate, ~85% of the time. This is likely because I limit my participation to Master workers and the job’s payment is within the minimum wage range. I also have a low dropout/incompletion rate on TurkPrime (~6%) compared to some of my MTurk studies (>12%). Some research suggests that workers prioritize fair compensation for HITs over interest in HITs – so it’s possible it’s not TurkPrime per se.
Though I didn’t add a gender filter, I had a comparable number of women and men for my two studies on TurkPrime (I’d love to hear more from others on their experience with this).
Open-ended feedback boxes were meaningful and not nonsensical.
Microbatching is a cost-effective feature that also reduces the possibility of time-of-day-related effects – This (in addition to others) is an important feature to ensures all HITs don’t get used up quickly, especially given the recent bot/server farm issue.
There is an easy-to-use feature that allows you to exclude participants from one study you’ve completed in a new study you’re doing.
Attrition rate that TurkPrime provides is super helpful.
TurkPrime FAQs and blogposts are very informative and accessible for newbies (MTurk can be a bit clunky for newbies).
Turkprime does, however, include additional demographic questions of their own, which as a Requester, I’m uncertain if the workers get paid to do.
In general, I find TurkPrime's interface less time consuming. I'd give it a higher rating than MTurk. To their credit, TurkPrime has done a great job of marketing the platform as being geared toward academics, so perhaps that in itself makes it more conducive for academic folks. So, yes, I'd choose TurkPrime over MTurk for my crowdsourced data :)
ETA (27/3/19): TurkPrime is expensive for data with not much ethnic/national diversity (though there is likely more political diversity). It's a quick shortcut for sure, but I think with better planning, you might get access to underrepresented populations by other means. I'd choose TurkPrime, but more likely as an additional study.