A Long-term VIS-enabled Infrastructure for Supporting ML-assisted Human Decision-making

许多大型组织拥有大量训练有素的人力资源库。当新的任务到来时，管理层通过选择具有不同技能的适当团队成员来构建团队，并为团队安排有效的运营结构。在机器学习(ML)中，模型开发人员通常为每个单独的任务训练许多模型，然后选择最佳模型来执行任务，同时丢弃未选择的模型。考虑到保留一个训练有素的ML模型的成本比雇佣一个人要少得多，这是对模型资源的巨大浪费。这种浪费做法背后的主要原因包括：(I)缺乏有效手段来理解大量ML模型的“技能概况”；(Ii)缺乏有效手段来构建一个“团队”，使得团队的综合技能适合于完成任务，但每个组成部分模型并不具备所需的所有技能；以及(Iii)缺乏有效手段使人类决策者能够将不完美的ML模型用作助手或顾问。由于这些原因，维持大量训练过的ML模型的动力较小，而这些模型对于特定的任务可能并不是最好的，重点一直放在为每个到达任务训练尽可能最佳的“明星”模型上。可视化和视觉分析(VIS)技术可以解决上述三个“不足”。在许多数据密集型应用中，可视化可以使决策者能够快速观察大量数据(例如股市)，分析不同数据实体之间的复杂关系(例如社会网络分析)，并基于多个有时相互冲突的机器预测(例如通过不同的流行病学模型)做出复杂的判断。最新的理论进步解释了可视化为用户提供了统计和算法无法提供的东西。人类在接收和推理信息方面的认知带宽有限。为了减少人类接收的信息量，统计和算法通常以更高的精度将大量数据转换为几个变量(例如，平均值和标准差)，而可视化以较低的精度呈现更多的变量(例如，时间序列中500个数据点的线图)。由于人类可以以非常低的认知成本直观地感知许多变量，因此更多的认知带宽可以引导到数据知情推理。这就解释了为什么金融专家很少只根据一两个财务指标做出决策，还需要观察时间序列数据。视觉分析是VIS的一个分支，专注于在人类决策工作流程中综合使用统计、算法、可视化和交互。在这个项目中，我们将开发一种新技术，使人类决策者能够在他们的工作流程中受益于VIS功能。我们通过设计和开发一种新型的支持VIS的基础设施来解决第一个问题，在该基础设施中，成百上千的ML模型可以连同它们的出处一起被存储，被自动和例行地测试和分析，并被ML模型开发人员作为训练有素的模型资源在VIS功能的帮助下进行管理。我们通过从模型资源池中选择适当的组件模型(即，团队成员)来为ML模型开发者提供用于构建集成模型(即，ML模型的团队)的VIS使能工具，并确定适当的集成策略(即，团队结构)来解决第二个问题。最后但并非最不重要的一点是，我们通过为ML模型用户(即从ML模型接收低级预测或建议的决策者)提供VIS功能，使他们能够快速观察不同模型做出的低级预测中的异常和冲突，并在有帮助的情况下仔细审查这些模型的概况和出处，从而解决第三个“缺失”问题。

Many large organisations maintains a large pool of trained human resources. When a new task arrives, the management constructs a team by selecting appropriate team members with different skills and arranges an effective operational structure for the team. In machine learning (ML), the model developers typically train many models for each individual task, then select the best model to perform the task, while discarding the unselected models. Considering that keeping a trained ML model costs much less than employing a person, there is a huge waste of model resources. The main reasons behind this wasteful practice include (i) the lack of effective means for apprehending the "skill profiles" a large number of ML models; (ii) the lack of effective means for constructing a "team" such that the combined skillset of the team is suitable for the task but each component model does not have all the skills required; and (iii) the lack of effective means for enabling human decision makers to utilise imperfect ML models as assistants or advisers. Because of these reasons, there is less incentive to maintain a large pool of trained ML models that may not be the best for a specific task individually, and the emphasis has been placed on training a "star" model as optimal as possible for each arrival task.The technology of visualization and visual analytics (VIS) can address the aforementioned three "lacks". In many data-intensive applications, VIS can enable decision-makers to observe a large amount of data quickly (e.g., stock market), analyse complex relationships among different data entities (e.g., social network analysis), and make complex judgement based on multiple and sometimes conflicting machine-predictions (e.g., by different epidemiological models). The latest theoretical advance offers an explanation as to what visualization offers users that statistics and algorithms cannot offer. Humans have limited cognitive bandwidth for receiving and reasoning about information. To reduce the amount of information received by humans, statistics and algorithms typically transform a large amount of data to a few variables (e.g., mean and standard deviation) at a higher precision, while visualization presents many more variables at a lower precision (e.g., a line plot of 500 data points in a time series). Because humans can perceive many variables visually at a very low cognitive cost, more cognitive bandwidth can be directed to data-informed reasoning. This explains why financial experts make decisions rarely based only on one or two financial indicators, but also need to observe time series data. Visual analytics is a branch of VIS focusing on combined uses of statistics, algorithms, visualization, and interaction in human decision workflows.In this project, we will develop a new technology to enable human decision makers to benefit from VIS capabilities in their workflows. We address the first aforementioned "lack" by designing and developing a novel VIS-enabled infrastructure where hundreds and thousands of ML models can be stored with their provenance, be tested and profiled automatically and routinely, and be managed as trained model resources by ML model-developers with the aid of VIS capabilities. We address the second "lack" by providing ML model-developers with a VIS-enabled tool for constructing ensemble models (i.e., teams of ML models) by selecting appropriate component models (i.e., team members) from a pool of model resources, and determine an appropriate ensemble strategy (i.e., team structure). Last but not least, we address the third "lack" by providing ML model users (i.e., decision makers who receive low-level predictions or recommendations from ML models) with the VIS capabilities, which allow them to observe quickly the anomalies and conflicts in the low-level predictions made by different models, and when it is helpful, to scrutinise the profile and provenance of these models.