Sparse, rank-reduced and general smooth modelling

当一些感兴趣的变量以复杂的方式与许多预测变量相关，并且我们想要理解这种关系时，平滑回归模型是有用的。在许多情况下，变量之间的依赖关系的复杂性意味着遵循传统的统计方法写下一个简单的统计模型来描述这种关系是不切实际的，其中只有几个未知参数需要估计。相反，统计模型是根据未知的平滑预测函数来指定的，例如“对数血压是由年龄的平滑函数加上体重和身高的平滑函数加上每周锻炼时间的平滑函数给出的”。统计上的挑战是如何估计光滑函数。经过几十年对这些平滑模型的理论和计算的研究，它们的应用现在已经很广泛，几乎和传统的回归模型一样常规。然而，在复杂的数据情况下，它们的使用仍然存在一些实际障碍，而在这些情况下它们应该最具有吸引力。目前的方法允许通过稀疏计算方法对短期空间、时间或时空相关性进行有效建模，或者通过降秩方法对涉及许多变量的复杂关系进行建模，但不能两者兼而有之。然而，具有短距离残差相关的复杂模型正是这种平滑模型最具实际吸引力的地方。在简化秩设置中，允许对高度复杂的模型进行可行的计算，到目前为止，最可靠和有效的计算方法仅限于感兴趣的变量来自指数分布族（正态，泊松，二项等）的情况。但是，考虑到这些方法的广泛应用，对于那些感兴趣的变量遵循与指数族非常不同的分布（例如，它可能是事件的等待时间，或者事件在空间位置的发生）的模型，也会有许多类似可靠的方法的应用。越来越多的研究人员和公司正在寻求分析非常大的数据集，这在当前的平滑建模技术下是根本不可行的。该项目旨在解决这些挑战，从而大规模地增加这类模型的实际范围和效用。特别是，该项目将寻求新的方法来混合稀疏和降秩方法来解决问题1；以指数族方法的经验为基础，从更一般的分布类别中开发可靠有效的变量方法，以解决2；并开发新颖有效的算法来处理大型复杂模型，这些模型可以在廉价的标准计算机硬件上轻松并行处理，以解决3。开发的方法将在免费的开源软件中实现，建立在PIs成功的mgcv包上，用于广义加性建模，在R统计计算环境中。这些方法也将通过教科书、短期课程和提供网络资源来传播。

Smooth regression models are useful when some variable of interest is related to a number of predictor variable in a complex manner, and we want to understand that relationship. In many cases the complexity of the dependence between the variables means that it is impractical to follow the traditional statistical approach of writing down a simple statistical model describing the relationship, in which only a few unknown parameters are to be estimated. Instead the statistical model is specified in terms of unknown smooth functions of predictors, for example `log blood pressure is given by a smooth function of age plus a smooth function of weight and height plus a smooth function of hours of exercise per week'. The statistical challenge is then to estimate the smooth functions. Given decades of work on the theory and computation of these smooth models, their use is now widespread and almost as routine as that of traditional regression models. However there remain several practical obstacles to their use, in exactly the complex data situations in which they should be most appealing.1. Current methods allow either the effective modelling of short range spatial, temporal or spatio-temporal correlation, via sparse computational methods, OR the modelling of complex relationships involving many variables, via reduced rank methods, but not both. However it is complex models with short range residual correlation are exactly where such smooth models are most practically appealing.2. In the reduced rank setting, that allows feasible computation with highly complex models, the most reliable and efficient computational methods are so far restricted to situations where variable of interest comes from the exponential family of distributions (normal, Poisson, binomial etc). But given the proven wide utility of such methods, there would also be many applications for similarly reliable methods for models where the variable of interest follows a very different distribution to those in the exponential family (for example it might be the waiting time to an event, or the occurrence of an event at a spatial location).3. Increasingly researchers and companies are seeking to analyze very large datasets, which are simply infeasible with current smooth modelling technology. This project aims to address these challenges, thereby massively increasing the practical scope and utility of this class of models. In particular the project will seek to find novel ways to hybridize the sparse and reduced rank approaches to smooth modelling to resolve issue 1; to build on experience with the exponential family methods to develop reliable and efficient methods for variables from a much more general class of distributions, to resolve 2; and to develop novel and efficient algorithms for handling large and complex models that can be readily parallelized on cheap standard computer hardware, to address 3. The methods developed will be implemented in free open source software, building on the PIs successfully mgcv package for generalized additive modelling, in the R statistical computing environment. The methods will also be disseminated via a textbook, short courses and the provision of web resources.