Comparison of manual, machine learning, and hybrid methods for video annotation to extract parental care data
Comparison of manual, machine learning, and hybrid methods for video annotation to extract parental care data
复制标题
DOI：
10.1111/jav.03167
复制
发表时间：
2023-12-14
影响因子：
1.7
通讯作者：
Schroeder，Julia
中科院分区：
文献类型：
--
作者：
Chan，Alex Hoi Hang;Liu，Jingqi;Schroeder，Julia
在野外测量父母的照顾行为是研究动物生态和进化的核心，但这通常是劳动和时间密集型的。最近出现了高效的开源工具，可以使用机器学习和计算机视觉技术从视频中量化动物的行为，但与传统方法相比，对这些工具的表现评估有限。为了深入了解不同的方法是如何从野外拍摄的视频中提取数据的，我们比较了从视频记录中估计的野生家禽麻雀的父母供给率。比较了专家手动标注、众包标注、基于开源软件DeepMeerkat的自动检测和混合标注四种方法。我们发现，自动方法收集的数据与专家注释(r = 0.62)相关，并进一步表明这些数据具有生物学意义，因为它们可以预测幼虫的存活率。然而，由于检测到非探视事件，自动方法产生的估计很大程度上是有偏见的，而众包和混合注解产生的估计相当于专家注解。与手动标注相比，混合标注方法需要大约20%的标注时间，这使其成为从视频收集数据的一种更具成本效益的方式。我们提供了一个成功的案例研究，说明如何使用预先存在的数据集来采用和评估不同的方法，以便就处理视频数据集的最佳方式做出明智的决策。如果现有的框架产生有偏见的估计，我们鼓励研究人员采用混合方法，首先使用机器学习框架对视频进行预处理，然后进行手动注释，以节省注释时间。随着开源机器学习工具变得越来越容易获得，我们鼓励生物学家利用这些工具来缩短注释时间，但仍然可以获得同样准确的结果，而不需要从头开始开发新的算法。
Measuring parental care behaviour in the wild is central to the study of animal ecology and evolution, but it is often labour‐ and time‐intensive. Efficient open‐source tools have recently emerged that allow animal behaviour to be quantified from videos using machine learning and computer vision techniques, but there is limited appraisal of how these tools perform compared to traditional methods. To gain insight into how different methods perform in extracting data from videos taken in the field, we compared estimates of the parental provisioning rate of wild house sparrowsPasser domesticusfrom video recordings. We compared four methods: manual annotation by experts, crowd‐sourcing, automatic detection based on the open‐source software DeepMeerkat, and a hybrid annotation method. We found that the data collected by the automatic method correlated with expert annotation (r = 0.62) and further show that these data are biologically meaningful as they predict brood survival. However, the automatic method produced largely biased estimates due to the detection of non‐visitation events, while the crowd‐sourcing and hybrid annotation produced estimates that are equivalent to expert annotation. The hybrid annotation method takes approximately 20% of annotation time compared to manual annotation, making it a more cost‐effective way to collect data from videos. We provide a successful case study of how different approaches can be adopted and evaluated with a pre‐existing dataset, to make informed decisions on the best way to process video datasets. If pre‐existing frameworks produce biased estimates, we encourage researchers to adopt a hybrid approach of first using machine learning frameworks to preprocess videos, and then to do manual annotation to save annotation time. As open‐source machine learning tools are becoming more accessible, we encourage biologists to make use of these tools to cut annotation time but still get equally accurate results without the need to develop novel algorithms from scratch.