Collaborative Research: ATD: Rapid Structure Recovery and Outlier Detection in Multidimensional Data

该项目旨在开发和应用来自代数的工具来识别嵌入在合法通信流和在线论坛中的攻击。该研究将加强统计方法和人工智能，以检测和打击威胁。具体来说，该研究将处理未标记的数据，并将适用于无法支持训练数据学习的环境，例如新领域或以前未见过的威胁。该项目将为本科生和研究生提供培训机会，为他们未来在STEM领域的职业生涯做好准备。对数据异常值检测、缺失数据恢复和隐藏约束检测的研究将在各个科学领域有广泛的应用。该项目旨在设计一种自适应线性时间算法来分离信号，寻找隐藏的约束方程，并检测高维数据（张量）中的相似性。这项由三位研究者和学生参与的合作研究将集中在三个独立的任务上。第一种方法将信号分离和离群值预测扩展到连续频谱。第二部分将重构代数结构为统一算法的张量网络。第三个将设计更快的（线性时间）矩阵系统的解决方案，以提高实际范围。对高维数据的分析经常与维数的诅咒相冲突：随着独立参数数量的增加，搜索邻居所需的时间呈指数增长。此外，异常值的含义变得模糊，因为距离和距离的概念难以区分，传统统计倾向于识别大的子空间。新的代数标记可以检测任何维度的结构，并且可以快速计算。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The project aims to develop and apply tools coming from algebra to identify attacks embedded within legitimate communications streams and online fora. The research will enhance statistical methods and Artificial Intelligence to detect and combat threats. Specifically, the research will work with unlabeled data and will be applicable in environments that cannot support learning on training data, such as new domains or previously unseen threats. The project will provide training opportunities for both undergraduate and graduate students, preparing them for their future careers in STEM fields. The research on detecting outliers in data, recovering missing data, and detecting hidden constraints will have many applications across the sciences.The project aims to design a self-adaptive linear-time algorithm to separate signals, find hidden constraint equations, and detect similarities in high-dimensional data (tensors). This collaborative research of the three investigators and student participants will focus on three independent tasks. The first will extend signal separation and outlier prediction to a continuous spectrum. The second will refactor algebraic structures into tensor networks for uniform algorithms. The third will devise faster (linear-time) solutions to matrix systems to enhance practical range. Analysis of high-dimensional data often runs afoul of the curse of dimensionality: as the number of independent parameters increases, the time needed to search neighbors grows exponentially. Also, the meaning of outlier becomes blurred as notions of far apart and close together are less distinguishable, and traditional statistics tend to identify large subspaces. The new algebraic markers will detect structure in any dimension and be quickly computable.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.