Collaborative Research: ATD: Rapid Structure Recovery and Outlier Detection in Multidimensional Data

该项目旨在开发和应用来自代数的工具，以识别嵌入合法通信流和在线论坛的攻击。这项研究将加强统计方法和人工智能，以检测和打击威胁。具体地说，这项研究将使用未标记的数据，并将适用于不支持学习训练数据的环境，例如新域或以前未见过的威胁。该项目将为本科生和研究生提供培训机会，为他们未来在STEM领域的职业生涯做好准备。数据中的离群点检测、缺失数据恢复和隐含约束检测的研究将在多个科学领域得到广泛应用。该项目旨在设计一种自适应线性时间算法来分离信号、找到隐含约束方程，并检测高维数据(张量)中的相似性。这项由三名研究人员和学生参与者共同进行的研究将集中在三项独立的任务上。第一种是将信号分离和离群值预测扩展到连续谱。第二种是将代数结构重构为用于统一算法的张量网络。第三种是设计矩阵系统更快的(线性时间)解决方案，以扩大实用范围。高维数据的分析经常与维度诅咒相冲突：随着独立参数的数量增加，搜索邻居所需的时间呈指数级增长。此外，离群值的含义变得模糊，因为相距较远和距离较近的概念不太容易区分，而且传统统计往往识别较大的子空间。新的代数标记将检测任何维度的结构，并可快速计算。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The project aims to develop and apply tools coming from algebra to identify attacks embedded within legitimate communications streams and online fora. The research will enhance statistical methods and Artificial Intelligence to detect and combat threats. Specifically, the research will work with unlabeled data and will be applicable in environments that cannot support learning on training data, such as new domains or previously unseen threats. The project will provide training opportunities for both undergraduate and graduate students, preparing them for their future careers in STEM fields. The research on detecting outliers in data, recovering missing data, and detecting hidden constraints will have many applications across the sciences.The project aims to design a self-adaptive linear-time algorithm to separate signals, find hidden constraint equations, and detect similarities in high-dimensional data (tensors). This collaborative research of the three investigators and student participants will focus on three independent tasks. The first will extend signal separation and outlier prediction to a continuous spectrum. The second will refactor algebraic structures into tensor networks for uniform algorithms. The third will devise faster (linear-time) solutions to matrix systems to enhance practical range. Analysis of high-dimensional data often runs afoul of the curse of dimensionality: as the number of independent parameters increases, the time needed to search neighbors grows exponentially. Also, the meaning of outlier becomes blurred as notions of far apart and close together are less distinguishable, and traditional statistics tend to identify large subspaces. The new algebraic markers will detect structure in any dimension and be quickly computable.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.