Collaborative Research: ATD: Rapid Structure Recovery and Outlier Detection in Multidimensional Data

该项目旨在开发和应用来自代数的工具，以识别嵌入在合法通信流和在线论坛中的攻击。该研究将加强统计方法和人工智能，以检测和打击威胁。具体而言，该研究将使用未标记的数据，并将适用于无法支持对训练数据进行学习的环境，例如新领域或以前未见过的威胁。该项目将为本科生和研究生提供培训机会，为他们未来在STEM领域的职业生涯做好准备。本项目旨在设计一种自适应线性时间算法来分离信号、寻找隐藏约束方程和检测高维数据（张量）中的相似性。这三名研究人员和学生参与者的合作研究将集中在三个独立的任务。第一个将扩展信号分离和离群值预测的连续频谱。第二部分将把代数结构重构为张量网络，以实现统一算法。第三个将设计更快的（线性时间）解决方案，以矩阵系统，以提高实用范围。高维数据的分析经常与维数灾难相冲突：随着独立参数数量的增加，搜索邻居所需的时间呈指数级增长。此外，离群值的含义变得模糊，因为远距离和近距离的概念难以区分，并且传统统计倾向于识别大型子空间。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

The project aims to develop and apply tools coming from algebra to identify attacks embedded within legitimate communications streams and online fora. The research will enhance statistical methods and Artificial Intelligence to detect and combat threats. Specifically, the research will work with unlabeled data and will be applicable in environments that cannot support learning on training data, such as new domains or previously unseen threats. The project will provide training opportunities for both undergraduate and graduate students, preparing them for their future careers in STEM fields. The research on detecting outliers in data, recovering missing data, and detecting hidden constraints will have many applications across the sciences.The project aims to design a self-adaptive linear-time algorithm to separate signals, find hidden constraint equations, and detect similarities in high-dimensional data (tensors). This collaborative research of the three investigators and student participants will focus on three independent tasks. The first will extend signal separation and outlier prediction to a continuous spectrum. The second will refactor algebraic structures into tensor networks for uniform algorithms. The third will devise faster (linear-time) solutions to matrix systems to enhance practical range. Analysis of high-dimensional data often runs afoul of the curse of dimensionality: as the number of independent parameters increases, the time needed to search neighbors grows exponentially. Also, the meaning of outlier becomes blurred as notions of far apart and close together are less distinguishable, and traditional statistics tend to identify large subspaces. The new algebraic markers will detect structure in any dimension and be quickly computable.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.