BRC-BIO: Optimizing Snake Identification by Understanding the Interplay of Computer Vision, Crowdsourcing, and Expert Verification

物种识别是生物学的基础，在蛇的情况下，不仅有助于了解它们的多样性，而且在蛇咬伤的情况下也很关键。最近，可以帮助人们完成分类任务的计算机系统变得越来越普遍。计算机视觉算法是经过训练的计算机程序，用于执行特定的分类任务，例如Shazam，Google透镜，iNaturalist的Seek和eBird的Merlin应用程序。本研究旨在了解冲突信息对分类任务准确性的影响。这很重要，因为它可以引导我们找到一个最佳的管道，让人类和计算机一起工作来执行这些任务，同时犯最少的错误。这个项目涉及从照片中识别蛇的种类，这对人类和计算机来说都是一项困难的任务，因为有4,000多种不同的蛇，其中一些看起来非常相似。在这个项目中，人们将看到蛇的图像，并要求他们识别它们的种类。有时，他们也会收到其他人或计算机视觉算法称之为蛇的信息。研究人员将测量由额外信息引起的识别准确性的任何变化。研究人员还将从自然历史博物馆保存的标本中收集稀有蛇类的图像，以确定这些类型的图像是否可以帮助训练计算机视觉算法，以更好地识别野生蛇类。该项目将为本科生和研究生提供培训机会，并邀请公众参与识别和数据收集。该项目将探索卷积神经网络（CNN）和人类物种识别的交叉点。它还将使用现场和收集数据研究来自不同训练集的CNN准确性的影响。使用游戏化平台，研究人员将预先测试参与者的蛇ID技能，然后随机分配参与者有或没有先前标签的蛇ID任务。一些先前的标签将是正确的，而另一些将是不正确的。一些先前的标签似乎来自其他参与者，另一些来自计算机视觉算法。研究人员将测量玩家提供与竞争或确认先前信息相匹配的错误ID的频率。玩家是否以及如何容易加入算法或其他人的不正确ID的“潮流”将有助于确定如何以及何时添加“人在回路”，同时保持或提高识别准确性。在第二个项目中，计算机视觉算法在博物馆标本，野生蛇，或两者都将进行比较，以便更好地了解使用博物馆标本的照片来滋养计算机视觉算法的训练数据集的潜力。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查进行评估，被认为值得支持的搜索.

Species identification is fundamental to biology, and in the case of snakes not only helps understanding their diversity but could also be critical in the case of snake bites. Recently, computer systems that can help people with classification tasks have become a lot more common. Computer vision algorithms are computer programs that have been trained to perform specific classification tasks, such as Shazam, Google Lens, iNaturalist’s Seek, and eBird’s Merlin app. This research aims to understand the influence that conflicting information can have on the accuracy of classification tasks. This is important because it could lead us to an optimal pipeline that allows humans and computers to work together to perform these tasks while making the minimal number of mistakes. This project involves the identification of snake species from photos, a task that is difficult for both humans and computers because there are over 4,000 different snake species and some of them look extremely similar. In the project, people will be shown images of snakes and asked to identify them to species. Sometimes, they will also receive information about what other people or computer vision algorithms called these snakes. The researchers will measure any changes in identification accuracy that are caused by the additional information. The researchers will also gather images of rare species of snakes from preserved specimens in natural history museums to determine whether these types of images can help train computer vision algorithms to better identify these snakes in the wild. This project will provide training opportunities for undergraduate and graduate students, as well as invite participation from the public in identification and data collection. This project will explore the intersection of Convolutional Neural Network (CNN) and human species identification. It will also study effects of accuracy of CNNs from different training sets, using field and collection data. Using a gamified platform, researchers will pre-test the snake ID skills of participants, then randomly assign participants snake ID tasks with or without prior labels. Some prior labels will be correct and others will be incorrect. Some prior labels will appear to come from other participants and others from computer vision algorithms. Researchers will measure how frequently players give an incorrect ID that matches competing or confirming prior information. Whether and how susceptible players are to join the “bandwagon” of incorrect IDs from algorithms or other humans will help determine how and when to add a “human-in-the-loop” while maintaining or increasing identification accuracy. In the second project, the performance of computer vision algorithms trained on images of museum specimens, wild snakes, or both will be compared in order to better understand the potential of using photos of museum specimens to nourish training datasets for computer vision algorithms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.