Mechanical cheat

Jim Giles and Zena Lovino's article "Scams, mischief and fraud in the crowd" (9th July, p20-21) describes some abuses of Amazon's Mechanical Turk by those offering online jobs. Abuse, however, also frequently occurs in the opposite direction. Researchers paying turkers to fill in multi-choice questionnaires find that a large proportion of responders game the system. In order to maximise their income, these gamers answer the questionnaire as quickly as possible by clicking the most convenient button. In order to exclude these misleading responses from their experimental data, researchers have to employ a variety of devices, such as planting a few questions with obvious answers then ignoring all answers from anyone that gets these planted questions wrong. Researchers who have not taken such precautions will find that referees reject their experimental results.