Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition.
Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition.
复制标题
DOI：
10.3390/jpm10030086
复制
发表时间：
2020-08-13
影响因子：
--
通讯作者：
Wall DP
中科院分区：
文献类型：
--
作者：
Washington P;Leblanc E;Dunlap K;Penev Y;Kline A;Paskov K;Sun MW;Chrisman B;Stockham N;Varma M;Voss C;Haber N;Wall DP
移动远程医疗正在成为精准健康和精准医疗的关键，甚至是必要的方面。在这项研究中，我们评估了一群虚拟工作者（定义为流行众包平台经过审查的成员）的能力和潜力，以帮助诊断自闭症。我们在众包任务时对工作人员进行评估，该任务为患有自闭症和神经正常对照的儿童的非结构化公共 YouTube 视频提供分类顺序行为评级。为了评估独立人群中一致的新兴模式，我们将两个众包平台上来自不同地理位置的工人作为目标：Amazon Mechanical Turk (MTurk) 上的国际工人群体 (N = 15) 和来自孟加拉国 (N = 56)、肯尼亚 (N = 23) 和菲律宾 (N = 25) 的微型工人。我们将工作人员的回答作为输入提供给经过验证的诊断机器学习分类器，该分类器是根据临床医生填写的电子健康记录进行训练的。我们发现，无论人群平台或目标国家如何，工作人员对分类器预测的正确诊断的平均置信度各不相同。根据先前的研究，最好的工人反应产生正确类别的平均概率高于 80%，超过一个标准差高于 50%，准确性和变异性与专家相当。花在任务上的平均时间与平均绩效之间存在弱相关性（r = 0.358，p = 0.005）。这些结果表明，虽然人群可以做出准确的诊断，但人群工作者评估行为特征的能力存在本质差异。我们提出了一种招募众包工作人员的新策略，以确保对自闭症以及许多其他潜在的儿科行为健康状况进行高质量的诊断评估。我们的方法代表了朝着基于人群的方法迈出的可行的一步，以实现更具可扩展性和负担得起的精准医疗。
Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workers—defined as vetted members of popular crowdsourcing platforms—to aid in the task of diagnosing autism. We evaluate workers when crowdsourcing the task of providing categorical ordinal behavioral ratings to unstructured public YouTube videos of children with autism and neurotypical controls. To evaluate emerging patterns that are consistent across independent crowds, we target workers from distinct geographic loci on two crowdsourcing platforms: an international group of workers on Amazon Mechanical Turk (MTurk) (N = 15) and Microworkers from Bangladesh (N = 56), Kenya (N = 23), and the Philippines (N = 25). We feed worker responses as input to a validated diagnostic machine learning classifier trained on clinician-filled electronic health records. We find that regardless of crowd platform or targeted country, workers vary in the average confidence of the correct diagnosis predicted by the classifier. The best worker responses produce a mean probability of the correct class above 80% and over one standard deviation above 50%, accuracy and variability on par with experts according to prior studies. There is a weak correlation between mean time spent on task and mean performance (r = 0.358, p = 0.005). These results demonstrate that while the crowd can produce accurate diagnoses, there are intrinsic differences in crowdworker ability to rate behavioral features. We propose a novel strategy for recruitment of crowdsourced workers to ensure high quality diagnostic evaluations of autism, and potentially many other pediatric behavioral health conditions. Our approach represents a viable step in the direction of crowd-based approaches for more scalable and affordable precision medicine.
登录
查看更多内容
影响因子：
7.5
作者：
Kalantarian, Haik;Jedoui, Khaled;Wall, Dennis P.
通讯作者：
Wall, Dennis P.
影响因子：
2.6
作者：
Gordon-Lipkin, Eliza;Foster, Jessica;Peacock, Georgina
通讯作者：
Peacock, Georgina
影响因子：
4.7
作者：
Iwabuchi SJ;Liddle PF;Palaniyappan L
通讯作者：
Palaniyappan L
影响因子：
5.9
作者：
Kalantarian H;Washington P;Schwartz J;Daniels J;Haber N;Wall DP
通讯作者：
Wall DP
DOI：
10.1145/3372300.3372308
发表时间：
2019-06-01
影响因子：
1
作者：
Kline, Aaron;Voss, Catalin;Wall, Dennis P.
通讯作者：
Wall, Dennis P.