SkyNet: an efficient and robust neural network training tool for machine learning in astronomy
SkyNet: an efficient and robust neural network training tool for machine learning in astronomy
复制标题
DOI：
10.1093/mnras/stu642
复制
发表时间：
2014-06-01
影响因子：
4.8
通讯作者：
Lasenby, Anthony
中科院分区：
文献类型：
--
作者：
Graff, Philip;Feroz, Farhan;Lasenby, Anthony
我们首次公开发布通用神经网络训练算法，称为 SkyNet。这种高效且强大的机器学习工具能够训练大型和深度的前馈神经网络，包括自动编码器，用于各种监督和无监督学习应用，例如回归、分类、密度估计、聚类和降维。 SkyNet 使用“预训练”方法来获得一组网络参数，这些参数经经验证明接近于良好的解决方案，然后使用牛顿法的正则化变体进行进一步优化，其中正则化的级别是自动确定和调整的；后者使用二阶导数信息来提高收敛性，但不需要评估或存储完整的Hessian矩阵，通过使用快速近似方法来计算Hessian向量乘积。这种方法的组合允许训练难以使用标准反向传播技术优化的复杂网络。 SkyNet 采用自然防止过度拟合的收敛标准，并且还包括用于估计网络输出准确性的快速算法。 SkyNet 的实用性和灵活性通过在许多玩具问题和天文问题中的应用得到了证明，这些问题侧重于从模糊和噪声图像中恢复结构、伽马射线暴的识别以及星系图像的压缩和去噪。 SkyNet 软件以标准 ANSI c 实现并使用 MPI 完全并行化，可从 http://www.mrao.cam.ac.uk/software/skynet/ 获取。
We present the first public release of our generic neural network training algorithm, called SkyNet. This efficient and robust machine learning tool is able to train large and deep feed-forward neural networks, including autoencoders, for use in a wide range of supervised and unsupervised learning applications, such as regression, classification, density estimation, clustering and dimensionality reduction. SkyNet uses a 'pre-training' method to obtain a set of network parameters that has empirically been shown to be close to a good solution, followed by further optimization using a regularized variant of Newton's method, where the level of regularization is determined and adjusted automatically; the latter uses second-order derivative information to improve convergence, but without the need to evaluate or store the full Hessian matrix, by using a fast approximate method to calculate Hessian-vector products. This combination of methods allows for the training of complicated networks that are difficult to optimize using standard backpropagation techniques. SkyNet employs convergence criteria that naturally prevent overfitting, and also includes a fast algorithm for estimating the accuracy of network outputs. The utility and flexibility of SkyNet are demonstrated by application to a number of toy problems, and to astronomical problems focusing on the recovery of structure from blurred and noisy images, the identification of gamma-ray bursters, and the compression and denoising of galaxy images. The SkyNet software, which is implemented in standard ANSI c and fully parallelized using MPI, is available at http://www.mrao.cam.ac.uk/software/skynet/.