III:Small: Outlier Discovery Paradigm

现代应用程序从金融交易系统、智能健康传感器和物联网设备收集的数据集数量惊人，其中分别包含从罕见现象到金融欺诈异常、健康警报到系统故障的关键见解。为了从赝品中破译出有价值的东西，分析师需要交互式地筛选和探索海量数据。通过发现异常，分析师可以检测金融欺诈，识别行为违规行为，或防止灾难性的传感器故障，从而以无数方式触及公民的生活。虽然存在用于检测特定类型离群值的独立算法的宝库，但它们往往是主题的变体。 这个研究项目是改变游戏规则的，因为它将提供第一个端到端的离群值服务，将这些丰富的算法带到集成的基础设施中，以支持有效的异常发现。该项目的更广泛影响还包括：PI的项目活动与STEM劳动力培训的整合;影响PI的WPI REU数据科学夏季网站;并影响PI领导和领导的数据科学新的跨学科学位课程，从博士，硕士到学士。PI在与各个级别的不同学生群体合作方面有着悠久的历史，并决心同样促进参与该项目的参与者的多样性。这项研究将远远超出开发另一种离群值检测算法，而是展示离群值发现作为服务的可行性。 它将从根本上开辟新的领域，支持离群点发现从识别，细化到解释。所提出的端到端异常发现范例将通过在一个集成平台内无缝集成异常相关服务来支持异常发现的所有阶段。其结果是一个受数据库系统启发的解决方案，它将服务建模为一等公民，以发现离群值。它将离群值检测过程与数据子间隔、离群值相对于其在原始数据集中的上下文的解释、关于域中离群值候选者的相关性的用户反馈以及用于改进离群值检测过程的有效性的度量学习相集成。 将使用离群基准数据集和与行业合作者合作探索的真实世界数据集和工作负载进行评估，以确定创新的效用。由此产生的系统将使分析师能够以人类的聪明才智引导发现过程，并在探索过程中通过平台的近实时交互式响应来增强能力。我们的解决方案旨在成为第一个通过将异常值解释服务和人类反馈整合到发现过程中来实现意义构建的力量的解决方案。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估而被认为值得支持。

Staggering volumes of data sets collected by modern applications from financial transaction systems, smart health sensors, and Internet of Things devices contain critical insights from rare phenomena to anomalies indicative of financial fraud, health alerts to system failure, respectively. To decipher the valuables from the counterfeit, analysts need to interactively sift through and explore the data deluge. By discovering anomalies, analysts may detect financial fraud, identify behavior irregularities, or prevent catastrophic sensor failures, thus touching the lives of citizens in countless ways. While a treasure trove of stand-alone algorithms for detecting particular types of outliers exists, they tend to be variations on a theme. This research project is game-changing in that it will offer the first end-to-end outlier services that bring this wealth of algorithms to bear in an integrated infrastructure to support effective anomaly discovery. The broader impact of this project also includes: the integration of the PI's project activities with the training of a STEM workforce; impacting the PI's WPI REU data science summer site; and impacting the new interdisciplinary degree programs from PhD, MS to BS in Data Science spearheaded and led by the PI. The PI has a long history of working with diverse student populations at all levels and is determined to similarly foster diversity of the participants involved in this project.This research will go well beyond developing yet another outlier detection algorithm by instead demonstrating the feasibility of outlier discovery as a service. It will break fundamentally new ground in supporting outlier discovery from identification, refinement to explanation. The proposed end-to-end anomaly discovery paradigm will support all stages of anomaly discovery by seamlessly integrating outlier-related services within one integrated platform. The result is a database-system inspired solution that models services as first class citizens for the discovery of outliers. It integrates outlier detection processes with data sub-spacing, explanations of outliers with respect to their context in the original data set, user feedback on the relevance of outlier candidates in the domain, and metric-learning to refine the effectiveness of the outlier detection process. Evaluation using outlier benchmark data sets and real-world data sets and workloads explored in partnerships with collaborators from industry will be conducted to establish the utility of the innovation. The resulting system will enable the analyst to steer the discovery process with human ingenuity, empowered by near real-time interactive responsiveness of the platform during exploration. Our solution aims to be the first to achieve the power of sense-making afforded by outlier explanation services and human feedback integrated into the discovery process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.