CIF: Small: Algorithms, Performance and Design for Sparsity-Enforced Learning

监督学习用于从一组训练数据中推断未知的回归函数。应用包括时间序列预测、遥感、医学图像分析、计算机视觉、人脸检测、文本分类、图像场景分类、语音识别和生物信息学。现有的学习算法有几个缺点。例如，基函数集通常包含未知参数，需要根据经验确定。由学习任务定义的优化问题依赖于需要仔细选择的权衡参数。现有学习方法的分析性能没有得到很好的研究，基函数集的作用也不清楚。此外，目前还没有一种有效的方法来选择基函数集。研究者开发了一种新的稀疏强制学习框架，用于回归函数学习，以克服现有方法的缺点。该框架包括用于稀疏向量估计的数值算法，用于性能分析的紧边界，以及用于字典设计的优化程序。推导函数的一种灵活形式是基函数的线性组合，这些基函数是通过离散未知参数来构造的。采用凸稀疏性增强、分层贝叶斯建模和归一化极大似然原理来学习权向量。离散化和稀疏性强制使最相关的基函数具有适当的参数自动选择。研究者通过推导学习权值的均方误差的紧界来分析学习框架的性能，特别是Cramer-Rao界和Hammersley-Chapman-Rob bins界。性能界限（及其简化）被用于优化设计过完备字典。研究者使用开发的框架来建模时间序列，并从图像中提取知识。

Supervised learning is used to infer an unknown regression function from a set of training data. Applications include time-series prediction, remote sensing, medical image analysis, computer vision, face detection, text categorization, image scene classification, speech recognition, and bioinformatics. Existing learning algorithms have several drawbacks. For example, the set of basis functions usually involves unknown parameters that need to be determined empirically. The optimization problem defined by the learning task depends on a trade-off parameter that requires careful selection. Analytical performance of the existing learning methods is not well studied and the role of the set of basis functions is not yet clear. In addition, currently there is no effective way to select the set of basis functions.The investigator develops a new framework of sparsity-enforced learning for regression function learning to overcome the drawbacks of existing approaches. This framework includes numerical algorithms for sparse vector estimation, tight bounds for performance analysis, and optimization procedures for dictionary design. A flexible form for the inferred function is a linear combination of basis functions, which are constructed by discretizing the unknown parameters. Algorithms are designed to learn the weight vector by convex sparsity enforcing, hierarchical Bayesian modeling and normalized maximum likelihood principle. The discretization and the sparsity enforcement enable automatic selection of most relevant basis functions with appropriate parameters. The investigator analyzes the performance of the learning framework by deriving tight bounds on the mean-squares error of the learned weights, in particular, the Cramer-Rao bound and Hammersley-Chapman-Rob bins bound. The performance bounds (and their simplifications) are employed to optimally design the overcomplete dictionary. The investigator uses the developed framework to model time series and extract knowledge from images.