A Long-term VIS-enabled Infrastructure for Supporting ML-assisted Human Decision-making

许多大型组织都拥有大量训练有素的人力资源。当新任务到来时，管理层通过选择具有不同技能的合适团队成员来构建团队，并为团队安排有效的操作结构。在机器学习（ML）中，模型开发人员通常为每个单独的任务训练许多模型，然后选择最佳模型来执行任务，同时丢弃未选择的模型。考虑到保持一个训练有素的ML模型的成本远低于雇用一个人，这是对模型资源的巨大浪费。这种浪费的做法背后的主要原因包括：(i)缺乏理解大量ML模型的“技能概况”的有效手段；（ii）缺乏构建“团队”的有效手段，使团队的综合技能集适合于任务，但每个组件模型都不具备所需的所有技能；（iii）缺乏有效的手段使人类决策者能够利用不完美的ML模型作为助手或顾问。由于这些原因，维护大量训练过的机器学习模型的动机就会减少，因为这些模型可能不是单个特定任务的最佳模型，并且重点放在为每个到达的任务训练尽可能最佳的“明星”模型上。可视化和可视化分析（VIS）技术可以解决上述三个“不足”。在许多数据密集型应用中，VIS可以使决策者快速观察大量数据（例如，股票市场），分析不同数据实体之间的复杂关系（例如，社会网络分析），并根据多个有时相互冲突的机器预测（例如，通过不同的流行病学模型）做出复杂的判断。最新的理论进展为可视化为用户提供了统计和算法无法提供的解释。人类接收和推理信息的认知带宽有限。为了减少人类接收到的信息量，统计学和算法通常以更高的精度将大量数据转换为几个变量（例如，均值和标准差），而可视化以更低的精度呈现更多变量（例如，在时间序列中有500个数据点的线形图）。由于人类可以以非常低的认知成本直观地感知许多变量，因此可以将更多的认知带宽用于数据知情推理。这就解释了为什么金融专家很少只根据一两个财务指标来做决策，他们还需要观察时间序列数据。可视化分析是VIS的一个分支，专注于统计、算法、可视化和人类决策工作流程中的交互的组合使用。在这个项目中，我们将开发一种新技术，使人类决策者能够从他们工作流程中的VIS功能中受益。我们通过设计和开发一种新的支持VIS的基础设施来解决前面提到的第一个“不足”，在这种基础设施中，成千上万的ML模型可以与其来源一起存储，自动和常规地进行测试和分析，并由ML模型开发人员在VIS功能的帮助下作为训练有素的模型资源进行管理。我们通过从模型资源池中选择适当的组件模型（即团队成员），并确定适当的集成策略（即团队结构），为ML模型开发人员提供一个支持vis的工具来构建集成模型（即ML模型团队），从而解决了第二个“不足”。最后但并非最不重要的是，我们通过为ML模型用户（即从ML模型接收低级预测或建议的决策者）提供VIS功能来解决第三个“不足”，这使他们能够快速观察不同模型所做的低级预测中的异常和冲突，并且在有帮助的时候，仔细检查这些模型的轮廓和来源。

Many large organisations maintains a large pool of trained human resources. When a new task arrives, the management constructs a team by selecting appropriate team members with different skills and arranges an effective operational structure for the team. In machine learning (ML), the model developers typically train many models for each individual task, then select the best model to perform the task, while discarding the unselected models. Considering that keeping a trained ML model costs much less than employing a person, there is a huge waste of model resources. The main reasons behind this wasteful practice include (i) the lack of effective means for apprehending the "skill profiles" a large number of ML models; (ii) the lack of effective means for constructing a "team" such that the combined skillset of the team is suitable for the task but each component model does not have all the skills required; and (iii) the lack of effective means for enabling human decision makers to utilise imperfect ML models as assistants or advisers. Because of these reasons, there is less incentive to maintain a large pool of trained ML models that may not be the best for a specific task individually, and the emphasis has been placed on training a "star" model as optimal as possible for each arrival task.The technology of visualization and visual analytics (VIS) can address the aforementioned three "lacks". In many data-intensive applications, VIS can enable decision-makers to observe a large amount of data quickly (e.g., stock market), analyse complex relationships among different data entities (e.g., social network analysis), and make complex judgement based on multiple and sometimes conflicting machine-predictions (e.g., by different epidemiological models). The latest theoretical advance offers an explanation as to what visualization offers users that statistics and algorithms cannot offer. Humans have limited cognitive bandwidth for receiving and reasoning about information. To reduce the amount of information received by humans, statistics and algorithms typically transform a large amount of data to a few variables (e.g., mean and standard deviation) at a higher precision, while visualization presents many more variables at a lower precision (e.g., a line plot of 500 data points in a time series). Because humans can perceive many variables visually at a very low cognitive cost, more cognitive bandwidth can be directed to data-informed reasoning. This explains why financial experts make decisions rarely based only on one or two financial indicators, but also need to observe time series data. Visual analytics is a branch of VIS focusing on combined uses of statistics, algorithms, visualization, and interaction in human decision workflows.In this project, we will develop a new technology to enable human decision makers to benefit from VIS capabilities in their workflows. We address the first aforementioned "lack" by designing and developing a novel VIS-enabled infrastructure where hundreds and thousands of ML models can be stored with their provenance, be tested and profiled automatically and routinely, and be managed as trained model resources by ML model-developers with the aid of VIS capabilities. We address the second "lack" by providing ML model-developers with a VIS-enabled tool for constructing ensemble models (i.e., teams of ML models) by selecting appropriate component models (i.e., team members) from a pool of model resources, and determine an appropriate ensemble strategy (i.e., team structure). Last but not least, we address the third "lack" by providing ML model users (i.e., decision makers who receive low-level predictions or recommendations from ML models) with the VIS capabilities, which allow them to observe quickly the anomalies and conflicts in the low-level predictions made by different models, and when it is helpful, to scrutinise the profile and provenance of these models.