A Long-term VIS-enabled Infrastructure for Supporting ML-assisted Human Decision-making

支持 ML 辅助人类决策的长期 VIS 基础设施

• 批准号: EP/X029557/1
• 负责人: Min Chen
• 金额: $ 74.45万
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2023
• 资助国家: 英国
• 起止时间: 2023 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FX029557%2F1
• 关键词: Long; term; VIS; enabled; Infrastructure

Many large organisations maintains a large pool of trained human resources. When a new task arrives, the management constructs a team by selecting appropriate team members with different skills and arranges an effective operational structure for the team. In machine learning (ML), the model developers typically train many models for each individual task, then select the best model to perform the task, while discarding the unselected models. Considering that keeping a trained ML model costs much less than employing a person, there is a huge waste of model resources. The main reasons behind this wasteful practice include (i) the lack of effective means for apprehending the "skill profiles" a large number of ML models; (ii) the lack of effective means for constructing a "team" such that the combined skillset of the team is suitable for the task but each component model does not have all the skills required; and (iii) the lack of effective means for enabling human decision makers to utilise imperfect ML models as assistants or advisers. Because of these reasons, there is less incentive to maintain a large pool of trained ML models that may not be the best for a specific task individually, and the emphasis has been placed on training a "star" model as optimal as possible for each arrival task.The technology of visualization and visual analytics (VIS) can address the aforementioned three "lacks". In many data-intensive applications, VIS can enable decision-makers to observe a large amount of data quickly (e.g., stock market), analyse complex relationships among different data entities (e.g., social network analysis), and make complex judgement based on multiple and sometimes conflicting machine-predictions (e.g., by different epidemiological models). The latest theoretical advance offers an explanation as to what visualization offers users that statistics and algorithms cannot offer. Humans have limited cognitive bandwidth for receiving and reasoning about information. To reduce the amount of information received by humans, statistics and algorithms typically transform a large amount of data to a few variables (e.g., mean and standard deviation) at a higher precision, while visualization presents many more variables at a lower precision (e.g., a line plot of 500 data points in a time series). Because humans can perceive many variables visually at a very low cognitive cost, more cognitive bandwidth can be directed to data-informed reasoning. This explains why financial experts make decisions rarely based only on one or two financial indicators, but also need to observe time series data. Visual analytics is a branch of VIS focusing on combined uses of statistics, algorithms, visualization, and interaction in human decision workflows.In this project, we will develop a new technology to enable human decision makers to benefit from VIS capabilities in their workflows. We address the first aforementioned "lack" by designing and developing a novel VIS-enabled infrastructure where hundreds and thousands of ML models can be stored with their provenance, be tested and profiled automatically and routinely, and be managed as trained model resources by ML model-developers with the aid of VIS capabilities. We address the second "lack" by providing ML model-developers with a VIS-enabled tool for constructing ensemble models (i.e., teams of ML models) by selecting appropriate component models (i.e., team members) from a pool of model resources, and determine an appropriate ensemble strategy (i.e., team structure). Last but not least, we address the third "lack" by providing ML model users (i.e., decision makers who receive low-level predictions or recommendations from ML models) with the VIS capabilities, which allow them to observe quickly the anomalies and conflicts in the low-level predictions made by different models, and when it is helpful, to scrutinise the profile and provenance of these models.

许多大型组织拥有大量训练有素的人力资源。当新任务到来时，管理层通过选择具有不同技能的合适团队成员来构建团队，并为团队安排有效的运营结构。在机器学习（ML）中，模型开发人员通常会为每个任务训练许多模型，然后选择最佳模型来执行任务，同时丢弃重复的模型。考虑到保持一个经过训练的ML模型的成本比雇用一个人要低得多，这是一个巨大的模型资源浪费。这种浪费的做法背后的主要原因包括：（i）缺乏有效的手段来理解大量ML模型的“技能配置文件”;（ii）缺乏有效的手段来构建一个“团队”，使得团队的组合技能适合任务，但每个组件模型并不具备所需的所有技能;（iii）缺乏有效的手段，使人类决策者能够利用不完美的ML模型作为助手或顾问。由于这些原因，没有太多的动机去维护一个大的训练ML模型池，这些模型可能不是最适合单个特定任务的，重点是训练一个“星星”模型，使其尽可能适合每个到达的任务。可视化和可视化分析技术（维斯）可以解决上述三个“缺乏”。在许多数据密集型应用中，维斯可使决策者快速观察大量数据（例如，股票市场），分析不同数据实体之间的复杂关系（例如，社交网络分析），并基于多个且有时相互冲突的机器预测（例如，不同的流行病学模型）。最新的理论进展解释了可视化为用户提供了统计和算法无法提供的东西。人类接受和推理信息的认知带宽有限。为了减少人类接收的信息量，统计和算法通常将大量数据转换为几个变量（例如，平均值和标准偏差），而可视化以较低的精度呈现更多的变量（例如，时间序列中500个数据点的线图）。由于人类可以以非常低的认知成本视觉地感知许多变量，因此可以将更多的认知带宽用于基于数据的推理。这就解释了为什么财务专家很少只根据一两个财务指标来做决策，还需要观察时间序列数据。可视化分析是维斯的一个分支，专注于在人类决策工作流程中综合使用统计、算法、可视化和交互。在本项目中，我们将开发一种新技术，使人类决策者能够在其工作流程中受益于维斯功能。我们通过设计和开发一个新的VIS支持的基础设施来解决上述第一个“缺乏”问题，在这个基础设施中，成千上万的ML模型可以与它们的出处一起存储，自动和常规地进行测试和分析，并由ML模型开发人员在维斯功能的帮助下作为经过训练的模型资源进行管理。我们通过为ML模型开发人员提供支持VIS的工具来构建集成模型（即，ML模型的团队）通过选择适当的组件模型（即，团队成员），并确定适当的整体策略（即，团队结构）。最后但并非最不重要的是，我们通过提供ML模型用户（即，从ML模型接收低级预测或建议的决策者）与维斯功能，这使他们能够快速观察不同模型所做的低级预测中的异常和冲突，并在有帮助时仔细检查这些模型的配置文件和出处。