A New Stochastic Neural Network: Statistical Perspectives and Applications

计算机技术与科学和日常生活的结合使科学家能够收集大量的数据。深度学习已经发展成为大数据分析的主要工具。然而，深度神经网络（DNN）的结构和参数很难解释，当将DNN应用于现实环境时，这可能会导致人机信任的严重问题。为了解决这一问题，研究人员在稀疏深度学习方面取得了相当大的进展，这证明了为潜在的非线性系统提供一致的相关变量选择。然而，由于DNN的黑箱性质，稀疏DNN的内部节点和参数仍然难以解释。研究者将开发一种新型的随机神经网络（StoNet），它是由许多简单回归组成的。随着训练样本量的增大，StoNet在函数逼近上与传统深度神经网络渐近等效，其结构和参数从统计角度更具可解释性。StoNet可以用来解决许多基本的统计任务，如非线性充分降维、因果推理、缺失数据和私人深度学习，这些都是传统深度神经网络难以处理的。StoNet通过其组合回归结构架起了线性模型和深度学习的桥梁，加深了人们对深度学习的理解。StoNet对可信赖的人工智能（AI）和数据驱动技术的发展具有潜在的巨大好处。研究结果将通过合作、出版物和会议报告传播给感兴趣的社区。该项目还将直接让研究生参与研究，并将研究成果纳入本科和研究生课程，对教育产生重大影响。与传统深度神经网络相比，StoNet为大数据分析提供了更通用、更强大的模型。它可以用来解决现代数据科学中经常遇到的许多基本统计任务。研究者将表明，通过在训练中对其层施加马尔可夫结构，StoNet产生了一种新颖的非线性充分降维方法。由此产生的方法具有可扩展性，可以处理比现有方法大得多的数据集。为了进行因果推理，研究者将开发一个因果关系网络（causal-StoNet）作为StoNet的变体，其中治疗变量作为网络中间层的可见单元。因果关系网络提供了一种方便的方法来建模结果函数和倾向评分，输入缺失数据，并识别高维问题的相关协变量。对于私人深度学习，研究者将开发一种变截断噪声随机梯度下降算法来训练StoNet。与现有的私有随机梯度下降算法相比，该算法在保证严格的差分隐私保证的同时，避免了梯度裁剪，提高了深度学习的收敛性和实用性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The integration of computer technology into science and daily life has enabled scientists to collect massive volumes of data. Deep learning has been developed as a major tool for big data analysis. However, the structure and parameters of the deep neural network (DNN) are hard to interpret, which can cause severe issues in human-machine trust when applying the DNN to real-life settings. To address this concern, researchers have made considerable progress in sparse deep learning, which provably leads to consistent selections of relevant variables for the underlying nonlinear system. However, the internal nodes and parameters of the sparse DNN are still hard to interpret due to the black-box nature of the DNN. The investigator will develop a new type of stochastic neural network (StoNet), which is a composition of many simple regressions. The StoNet is asymptotically equivalent to the conventional DNN in function approximation as the training sample size becomes large, while its structure and parameters are more interpretable from statistical perspectives. The StoNet can be employed to address many fundamental statistical tasks such as nonlinear sufficient dimension reduction, causal inference, missing data, and private deep learning that are difficult to handle with the conventional DNN. The StoNet bridges linear models and deep learning by its compositional regression structure, which deepens people’s understanding of deep learning. The StoNet has potentially immense benefits to the development of trustworthy artificial intelligence (AI) and data driven technologies. The research results will be disseminated to communities of interest via collaborations, publications, and conference presentations. The project will also have significant impacts on education by directly involving graduate students in the research and incorporating the research results into undergraduate and graduate courses. The StoNet provides a more general and powerful model for big data analysis than the conventional DNN. It can be employed to address many fundamental statistical tasks that are frequently encountered in modern data science. The investigator will show that the StoNet results in a novel nonlinear sufficient dimension reduction method by imposing a Markovian structure on its layers in training. The resulting method is scalable and can deal with much larger datasets than can the existing methods. For causal inference, the investigator will develop a causal-StoNet as a variant of StoNet, where the treatment variable is included as a visible unit in a middle layer of the network. The causal-StoNet provides a convenient way for modeling the outcome function and propensity score, imputing missing data, and identifying relevant covariates for high-dimensional problems. For private deep learning, the investigator will develop a varying truncation noisy stochastic gradient descent algorithm for training the StoNet. Compared to the existing private stochastic gradient descent algorithms, the proposed algorithm avoids gradient clipping and improves convergence and utility of deep learning while ensuring rigorous differential privacy guarantees.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.