Machine learning-based vision for "green-on-green'' spraying

智能喷洒的目标是更精确地定位除草剂。这减少了浪费，有利于环境。这种喷洒的一个关键步骤是识别杂草。典型的方法是使用计算机视觉，通常是基于机器学习的方法，对从上面观察杂草和作物的相机拍摄的照片进行操作。目前的视觉技术已经被证明能够处理“绿色对棕色”的场景，在杂草和作物很容易在非常明显的颜色背景（如土壤）上发现的情况下，对杂草的检测具有很高的准确性。在作物和杂草生长的早期阶段，这就足够了。然而，在生长的后期，作物和杂草的冠层开始重叠，准确有效地检测杂草变得更加困难。这种“绿色对绿色”的情况目前超出了可行的处理范围。解决“绿色对绿色”的杂草检测问题是本博士的重点。“绿对绿”之所以难，是因为我们不能依靠简单的色彩分割。在“绿上棕”中，将图像分割为绿色和棕色区域，可以识别出形状独特的绿色区域，并且易于分类。无论是基于经典机器视觉还是更现代的深度学习方法，现有的检测器都是如此。当植物重叠时，绿色区域不再包含如此独特的形状，或者如此大面积的独特形状，现有的检测方法因此而挣扎。答案是建造探测器来寻找除了颜色以外的东西。该博士将在基于深度学习的视觉框架下，追求两条探究线：为图像数据添加额外的维度，以及构建专注于不同特征的检测器。1)增加维度。传统的成像使用RGB相机，它在3个宽的视觉波段（我们熟悉的红、绿、蓝波段）报告强度。我们将研究额外数据层的使用。我们将探讨的一种可能性是使用RGB-D相机生成的深度数据。这些相机在每个像素中报告到物体的距离，实际上给出了关于树冠轮廓的信息。我们还将研究多光谱相机的使用，它提供了一个额外的光谱带，即“红边”。这个波段介于RGB相机通常的红色波段和近红外波段之间，是植被反射率（由相机测量）变化很大的区域，这意味着“红边”波段往往是非常有用的。2)关注不同的特点。最近基于机器学习的视觉研究已经开始关注检测器，实际上，检测器是一个集合，其中每个成员在图像中寻找不同的元素。一个典型的例子是使用在不同大小的对象上训练的集成成员，它生成的检测器在某种程度上是尺度不变的。我们将专注于识别与杂草检测相关的特征，训练检测器识别这些特征，然后将这些检测器组合起来。检测的准确性虽然很重要，但并不是唯一的考虑因素。我们开发的方法还需要运行速度足够快，以便能够在以15公里/小时的速度移动的喷雾器上部署。一般来说，当使用基于机器学习的方法时，在可以获得的精度和可以处理图像的速度之间存在权衡。这种权衡可以在多个维度上进行管理，例如图像大小、图像处理主干的深度和检测头的复杂性。我们将对所考虑的各种方法进行权衡。在此过程中，我们将考虑Riseholme设置中可用的各种处理选项、机载喷雾器处理、使用5G设置的边缘处理、云处理以及其他选项（如FGPA）。

The goal of intelligent spraying is that herbicides are more precisely targeted. This reduces waste and is beneficial for the environment. A key step in such spraying is identifying weeds. Typical approaches to doing this use computer vision, typically methods based around the use of machine learning, operating on pictures taken from cameras that view weeds and crops from above.Current vision technology has proved able to handle "green-on-brown'' scenarios, producing good accuracy of detection of weeds where weeds and crops are easy to spot against very distinctly coloured backgrounds, such as soil. This is sufficient in the early stages of growth, when crops and weeds are small. However, in later stages of growth, and the canopies of crops and weeds begin to overlap, accurately and efficiently detecting weeds becomes much harder. This "green-on-green'' scenario is currently beyond what can feasibly be handled. Solving the "green-on-green'' weed detection problem is the focus of this PhD.The reason that "green-on-green'' is hard, is that we cannot rely on simple colour segmentation. In the "green-on-brown'', segmenting images into green and brown areas identifies green areas with distinctive shapes that can easily be classified. This is what is going on in existing detectors whether they are based on classical machine vision or more modern deep learning approaches. When plants overlap, the green regions no longer contain such distinctive shapes, or such large areas of distinctive shapes, and existing approaches to detection struggle as a result. The answer is to build detectors that look for things other than just colour. This PhD will pursue two lines of inquiry: adding additional dimensions to the image data, and building detectors that focus on different features, in the framework of deep learning-based vision.1) Adding dimensions. Conventional imaging uses RGB cameras which report intensity in 3 wide visual bands (the familiar red, green and blue bands). We will investigate the use of additional layers of data. One possibility that we will look at is the use of depth data generated by RGB-D cameras. These cameras report the distance to the object in each pixel, giving, in effect, information about the contours of the canopy. We will also look at the use of multispectral cameras, which provide an additional spectral band, the "red edge''. This band is between the usual red band of RGB cameras and near infra-red, and is a region in which the reflectance (what is measured by a camera) of vegetation changes a lot, meaning that the "red edge'' band tends to be very informative.2) Focusing on different features. Recent research in machine learning-based vision has begun to focus on detectors that are, in effect, ensembles where each member looks for different elements in an image. A typical example is to use ensemble members that are trained on objects of different size, that generates detectors that are somewhat scale invariant. We will focus on identifying features that are relevant for weed detection training detectors to identify just these features, and then combining these detectors.Accuracy of detection, while important, is not the only consideration. The methods we develop also need to run fast enough that they can be deployed on a sprayer moving at 15km/h. In general, when using machine learning-based approaches, there is a tradeoff between the accuracy that can be obtained, and the speed with which images can be processed. This tradeoff can be managed along a number of dimensions, such as image size, depth of image processing backbone, and the complexity of the detection head. We will examine this tradeoff for the various approaches that we consider. In doing this, we will take into account various options for processing that are available as part of the setup at Riseholme, processing on-board the sprayer, edge-processing using the 5G setup, and cloud processing, and other options such as FGPA.