Challenges of data collection on mturk: A human-AI joint face-matching task

Michelle Mancenido, Pouria Salehi, Erin Chiou, Ahmadreza Mosallanezhad, Aksheshkumar Shah, Myke Cohen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations


During the COVID-19 pandemic, many human-subject studies have stopped in-person data collection and shifted to virtual platforms like Amazon Mechanical Turk (MTurk). This shift involves important considerations for study design and data analysis, particularly for studies involving behavioral assessment and performance with technology. We report on lessons learned from a recent study that used MTurk for a face-matching task with an open-source AI. Participants received $5 compensation for completing a 45-minute session that included questionnaires. To help address data validity issues, Qualtrics fraud-detection features (i.e., reCAPTCHA, ID-Fraud), trap-items (e.g., Respond with Often), and a modified-batch-randomization-process were employed. Participants' accumulative accuracy and response rates were also assessed. Out of 272 participants, 121 passed the data inclusion criteria. The questionnaires' reliability was within range (average 0.78) for the healthy dataset. Accumulative accuracy in the face-matching task decreased approximately halfway through the task. Subsequent data inspection revealed that almost half of the participants spent longer than 20 seconds and up to 12 minutes on a random image pair. It is possible that participants were interrupted during the study or they elected to take unscheduled breaks. Environmental factors that were easier to control during in-person laboratory studies now require built-in controls for virtual study environments. We learned that: (1) it is imperative to monitor performance measures over time for each participant; (2) the study duration may need to be kept shorter on virtual platforms compared to in-person studies; (3) an optional, planned break during the task might help prevent other unplanned breaks.

Original languageEnglish (US)
Title of host publicationIISE Annual Conference and Expo 2021
EditorsA. Ghate, K. Krishnaiyer, K. Paynabar
PublisherInstitute of Industrial and Systems Engineers, IISE
Number of pages6
ISBN (Electronic)9781713838470
StatePublished - 2021
EventIISE Annual Conference and Expo 2021 - Virtual, Online
Duration: May 22 2021May 25 2021

Publication series

NameIISE Annual Conference and Expo 2021


ConferenceIISE Annual Conference and Expo 2021
CityVirtual, Online


  • Crowdsourcing
  • Face verification
  • Human-AI joint decision systems

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Industrial and Manufacturing Engineering


Dive into the research topics of 'Challenges of data collection on mturk: A human-AI joint face-matching task'. Together they form a unique fingerprint.

Cite this