Citizen crowds and experts: observer variability in image-based plant phenotyping.
Citizen crowds and experts: observer variability in image-based plant phenotyping.
复制标题
DOI：
10.1186/s13007-018-0278-7
复制
发表时间：
2018
期刊：
影响因子：
5.1
通讯作者：
Tsaftaris SA
中科院分区：
文献类型：
--
作者：
Giuffrida MV;Chen F;Scharr H;Tsaftaris SA
基于图像的植物表型分析已成为揭示基因型与环境相互作用的有力工具。图像分析和机器学习的利用在提取来自表型实验的数据方面变得至关重要。然而，我们依赖于观察者（人类专家）的输入来执行表型分析过程。我们假设这样的输入是一个“黄金标准”，并使用它来评估软件和算法，并训练基于学习的算法。然而，我们应该考虑在有经验的和无经验的（包括普通公民）观察者之间是否存在任何差异。在这里，我们设计了一个研究，测量这样的变异性的注释任务的一个整数可量化的表型：叶数。我们比较了几个有经验和无经验的观察员在注释叶数的图像拟南芥测量内和观察员之间的变异性在一个控制的研究中使用专门设计的注释工具，但也公民使用分布式公民供电的基于网络的平台。在对照研究中，观察者通过观察用低分辨率和高分辨率光学器件拍摄的顶视图图像来计数叶子。我们评估了专门为此任务设计的工具的利用是否有助于减少这种变化。我们发现，工具的存在有助于减少观察者内的变异性，虽然观察者内和观察者间的变异性是存在的，但它对纵向叶数趋势统计评估没有任何影响。我们比较了公民提供的注释（来自基于网络的平台）的可变性，发现普通公民可以提供统计上准确的叶计数。我们还比较了最近的基于机器学习的叶子计数算法，发现虽然性能接近，但它仍然不在观察者之间的变化范围内。虽然观察者的专业知识发挥了作用，但如果存在足够的统计能力，则可以将一组无经验的用户甚至公民包括在基于图像的表型注释任务中，只要它们被适当地设计。我们希望通过这些发现，我们可以重新评估我们对自动算法的期望：只要它们的性能在观察者的变化范围内，它们就可以被认为是合适的替代方案。此外，我们希望激发人们对在公民驱动的平台上引入适当设计的任务的兴趣，不仅要获得有用的信息（用于研究），还要帮助公众参与这个社会重要问题。
Image-based plant phenotyping has become a powerful tool in unravelling genotype–environment interactions. The utilization of image analysis and machine learning have become paramount in extracting data stemming from phenotyping experiments. Yet we rely on observer (a human expert) input to perform the phenotyping process. We assume such input to be a ‘gold-standard’ and use it to evaluate software and algorithms and to train learning-based algorithms. However, we should consider whether any variability among experienced and non-experienced (including plain citizens) observers exists. Here we design a study that measures such variability in an annotation task of an integer-quantifiable phenotype: the leaf count. We compare several experienced and non-experienced observers in annotating leaf counts in images of Arabidopsis Thaliana to measure intra- and inter-observer variability in a controlled study using specially designed annotation tools but also citizens using a distributed citizen-powered web-based platform. In the controlled study observers counted leaves by looking at top-view images, which were taken with low and high resolution optics. We assessed whether the utilization of tools specifically designed for this task can help to reduce such variability. We found that the presence of tools helps to reduce intra-observer variability, and that although intra- and inter-observer variability is present it does not have any effect on longitudinal leaf count trend statistical assessments. We compared the variability of citizen provided annotations (from the web-based platform) and found that plain citizens can provide statistically accurate leaf counts. We also compared a recent machine-learning based leaf counting algorithm and found that while close in performance it is still not within inter-observer variability. While expertise of the observer plays a role, if sufficient statistical power is present, a collection of non-experienced users and even citizens can be included in image-based phenotyping annotation tasks as long they are suitably designed. We hope with these findings that we can re-evaluate the expectations that we have from automated algorithms: as long as they perform within observer variability they can be considered a suitable alternative. In addition, we hope to invigorate an interest in introducing suitably designed tasks on citizen powered platforms not only to obtain useful information (for research) but to help engage the public in this societal important problem.
登录
查看更多内容
影响因子：
5.1
作者：
Laskin, David N.;McDermid, Gregory J.
通讯作者：
McDermid, Gregory J.
影响因子：
4.5
作者：
Bock, C. H.;Parker, P. E.;Gottwald, T. R.
通讯作者：
Gottwald, T. R.
影响因子：
56.9
作者：
Berardini, TZ;Bollman, K;Poethig, RS
通讯作者：
Poethig, RS
DOI：
10.1109/tcbb.2015.2404810
发表时间：
2015-11-01
影响因子：
4.5
作者：
Dellen, Babette;Scharr, Hanno;Torras, Carme
通讯作者：
Torras, Carme
影响因子：
1.9
作者：
Hartung, K.;Piepho, H. -P.
通讯作者：
Piepho, H. -P.