III:Small: Outlier Discovery Paradigm

现代应用程序从金融交易系统、智能健康传感器和物联网设备中收集的惊人数量的数据集分别包含从罕见现象到指示金融欺诈的异常、健康警报到系统故障的关键见解。为了从假货中分辨出有价值的东西，分析师需要交互式地筛选和探索海量数据。通过发现异常，分析师可以发现金融欺诈，识别行为违规，或防止灾难性的传感器故障，从而以无数种方式触及公民的生活。虽然存在大量用于检测特定类型异常值的独立算法，但它们往往是同一主题的变体。这个研究项目改变了游戏规则，因为它将提供首个端到端异常值服务，将丰富的算法应用到集成基础设施中，以支持有效的异常发现。该项目更广泛的影响还包括：PI的项目活动与STEM劳动力培训的整合；影响PI的WPI REU数据科学夏季站点；并影响了由PI牵头的数据科学博士、硕士和学士等新的跨学科学位课程。PI在与不同层次的学生群体合作方面有着悠久的历史，并决心同样促进参与该项目的参与者的多样性。这项研究将远远超出开发另一种异常点检测算法，而是证明异常点发现作为一种服务的可行性。它将在支持从识别、提炼到解释的异常值发现方面，从根本上开辟新的领域。提出的端到端异常发现范式将通过在一个集成平台内无缝集成离群相关服务来支持异常发现的所有阶段。其结果是一个受数据库系统启发的解决方案，将服务建模为发现异常值的一等公民。它将异常点检测过程与数据子间距、原始数据集中异常点上下文的解释、用户对领域中异常点候选相关性的反馈以及度量学习集成在一起，以改进异常点检测过程的有效性。将使用离群基准数据集和与行业合作者合作探索的真实数据集和工作负载进行评估，以确定创新的效用。由此产生的系统将使分析人员能够通过人类的聪明才智来引导发现过程，并在探索过程中通过平台的近实时交互响应来增强其能力。我们的解决方案旨在成为第一个通过将异常值解释服务和人类反馈集成到发现过程中来实现意义构建能力的解决方案。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Staggering volumes of data sets collected by modern applications from financial transaction systems, smart health sensors, and Internet of Things devices contain critical insights from rare phenomena to anomalies indicative of financial fraud, health alerts to system failure, respectively. To decipher the valuables from the counterfeit, analysts need to interactively sift through and explore the data deluge. By discovering anomalies, analysts may detect financial fraud, identify behavior irregularities, or prevent catastrophic sensor failures, thus touching the lives of citizens in countless ways. While a treasure trove of stand-alone algorithms for detecting particular types of outliers exists, they tend to be variations on a theme. This research project is game-changing in that it will offer the first end-to-end outlier services that bring this wealth of algorithms to bear in an integrated infrastructure to support effective anomaly discovery. The broader impact of this project also includes: the integration of the PI's project activities with the training of a STEM workforce; impacting the PI's WPI REU data science summer site; and impacting the new interdisciplinary degree programs from PhD, MS to BS in Data Science spearheaded and led by the PI. The PI has a long history of working with diverse student populations at all levels and is determined to similarly foster diversity of the participants involved in this project.This research will go well beyond developing yet another outlier detection algorithm by instead demonstrating the feasibility of outlier discovery as a service. It will break fundamentally new ground in supporting outlier discovery from identification, refinement to explanation. The proposed end-to-end anomaly discovery paradigm will support all stages of anomaly discovery by seamlessly integrating outlier-related services within one integrated platform. The result is a database-system inspired solution that models services as first class citizens for the discovery of outliers. It integrates outlier detection processes with data sub-spacing, explanations of outliers with respect to their context in the original data set, user feedback on the relevance of outlier candidates in the domain, and metric-learning to refine the effectiveness of the outlier detection process. Evaluation using outlier benchmark data sets and real-world data sets and workloads explored in partnerships with collaborators from industry will be conducted to establish the utility of the innovation. The resulting system will enable the analyst to steer the discovery process with human ingenuity, empowered by near real-time interactive responsiveness of the platform during exploration. Our solution aims to be the first to achieve the power of sense-making afforded by outlier explanation services and human feedback integrated into the discovery process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.