Understanding the annotation process: annotation for Big data

数据的收集和创建速度是人类历史上最快的；到目前为止，其中绝大多数是数字格式的。与此相结合，以前“离线”的信息现在可以迅速和廉价地数字化，例如旧手稿、地图等。这些现有的和新的信息的海量收集创造了新的机会，也带来了困难。为了让这些信息中的许多有用，必须以某种方式对其进行分类和注释，以便能够理解数据，也可以更容易地访问正确的数据。用人工注释员手工完成这一分类是可能的，但这一努力可能会耗费时间、金钱和资源。对于大型数据集或需要专门知识来注释的数据集来说，情况尤其如此。考虑到这一成本，许多人转向机器学习来注释数据；然而，机器学习方法仍然需要人工干预来为算法创建训练集和判断算法的输出。因此，在分类和注释过程的某个阶段，不可避免地需要人为干预。在这个项目中，我们的目标是更好地了解这一注释过程，以便我们能够提供指导方针、方法和过程，为数据集提供最具成本效益和准确的注释。我们建议使用大数据中面临的三种主要类型的非结构化数据：文本、图像和视频。第一个挑战是更好地理解评价员在注释和评判不同类型的材料时所经历的过程。这将使用定性和定量技术的混合，使用较小规模的基于实验室的研究。通过更好地理解个人对材料进行注释和分类的过程，我们希望提供可用于使注释过程更有效的见解，并确定一组影响注释性能的初始因素，例如领域专业知识的程度和时间。在这项初步工作的基础上，目标是使用大规模众包风格的方法，调查这些因素中哪些对评估影响最大。最后一个挑战与分类任务有关：应该如何处理注释，以便在机器学习中使用时获得最佳结果？在此基础上，该项目旨在为创建注释和相关集制定一套指导方针。

Data is being collected and created at the fastest rate in human history; by the far the vast majority of this is in digital format. Allied with this, what was previously "offline" information can now be digitised quickly and cheaply e.g. old manuscripts, maps etc. This vast collection of existing and new information creates new opportunities and also difficulties. For a lot of this information to be useful it must be categorised and annotated in some way, so that sense can be made of the data and also so that the correct data can be accessed more easily. It is possible to complete this categorisation by hand with human annotators, but this effort can be expensive in terms of time, money and resources. This is especially true for large data sets or for data sets that require niche expertise to annotate. With this expense in mind, many have turned to machine learning to annotate data; however machine learning approaches still require human intervention to both create training sets for algorithms and judge the output of algorithms. Thus it is inevitable that human intervention is involved at some stage of the categorisation and annotation process. In this project we aim to gain a better understanding of this annotation process so that we can provide guidelines, approaches and processes for providing the most cost effective and accurate annotations for data sets. We propose to work with the three main types of unstructured data faced in big data: text, image, and video. The first challenge is to better understand the process assessors go through when annotating and judging different types of material. This will be carried out using a mixture of qualitative and quantitative techniques, using smaller scale lab-based studies. By better understanding the process by which individuals annotate and classify material, we hope to provide insights which can be used to make the annotation process more efficient, and identify a set of initial factors which affect annotation performance, such as degree of domain expertise and time. Based on this initial work, the aim is to then investigate which of these factors most affect assessment, using large scale crowdsourcing style methods. The final challenge is related to the classification task: how should annotation be approached, to give the best results when used in machine learning? Based on this, the project aims to create a set of guidelines for the creation of annotation and relevance sets.