Collaborative Research: Statistical Properties of Numerical Derivatives and Algorithms

Numerical differentiation is widely used in econometrics and many other areas of quantitative economic analysis. Many functions that need to be differentiated in econometric analysis need to be estimated from the data. For example, estimating the approximate variance of an estimator often requires estimating the derivatives of the moment conditions that define the estimator. Many estimators are also obtained by finding the zeros of the first order condition of the sample objective functions.The estimated functions can be either non-differentiable or difficult to differentiate analytically. Oftentimes the estimated functions are complex and can be challenging to compute even numerically. Empirical researchers often apply numerical differentiation methods which depend on taking a finite number of differences of the objective function at discrete points, either explicitly or implicitly through the use of software routines, to the estimated functions from the sample in order to approximate the derivative of the unknown true functions.A key tuning parameter that determines how well the numerical derivatives approximate the analytic derivatives is the step size used in the finite differencing operation. Empirical researchers often find that different step sizes can lead to very different numerical derivative estimates. Whilethe importance of numerical derivatives has not gone unnoticed in econometrics, statistics and mathematics, the results that are available in the existing literature are very limited in scope.The goal of this project is to take an important step to provide a systematic framework for understanding the conditions on the step size in numerical di^erentiation that are needed to obtain the optimal quality of approximation. These conditions involve subtle tradeoffs between the complexity of the function that needs to be differentiated and the amount of information that is available in the sample of data, and the degree of smoothness of the expectation of the function with respect to the sampling distribution. Empirical process theory provides a powerful tool for analyzing the complex of functions in the presence of randomly sampled data.This project focuses on analyzing the use of numerical derivatives in estimating the asymptotic variance of estimators and in obtaining extreme estimators through gradient based optimization routines. The PIs' first goal is to give general sufficient consistency conditions that allow for nondifferentiable and discontinuous moment functions in consistent variance estimation. The precise rate conditions for the step size in numerical differentiation that we obtain depend on the tradeoff between bias and the degree of nonsmoothness of the moment condition. These general conditionscan be specialized for certain continuous models, for which choosing a smaller step size can only be beneficial in reducing the asymptotic bias. However, the asymptotic bias will be dominated by the statistical noise once it falls below a certain threshold. The second goal of this project is to analyze a class of estimators that are based on numerically differentiating a finite sample objective function, and provide conditions under which numerical derivative based optimization methods deliver consistent and asymptotic normal parameter estimates. The conditions for numerical extreme estimators require that the step size used in thenumerical derivative has to converge to zero at specific rates when the sample size increases to infinity. The conditions required for the consistency of the asymptotic variance and for the convergence of the estimator itself can be different. The PIs seek extensive results that cover finite dimensionalparametric models, infinite dimensional semiparametric models, and models that are defined by U-processes involving multiple layers of summation over the sampling data. The proposed project involves joint work with Professor Aprajit Mahajan from Stanford University.

Numerical differentiation is widely used in econometrics and many other areas of quantitative economic analysis. Many functions that need to be differentiated in econometric analysis need to be estimated from the data. For example, estimating the approximate variance of an estimator often requires estimating the derivatives of the moment conditions that define the estimator. Many estimators are also obtained by finding the zeros of the first order condition of the sample objective functions.The estimated functions can be either non-differentiable or difficult to differentiate analytically. Oftentimes the estimated functions are complex and can be challenging to compute even numerically. Empirical researchers often apply numerical differentiation methods which depend on taking a finite number of differences of the objective function at discrete points, either explicitly or implicitly through the use of software routines, to the estimated functions from the sample in order to approximate the derivative of the unknown true functions.A key tuning parameter that determines how well the numerical derivatives approximate the analytic derivatives is the step size used in the finite differencing operation. Empirical researchers often find that different step sizes can lead to very different numerical derivative estimates. Whilethe importance of numerical derivatives has not gone unnoticed in econometrics, statistics and mathematics, the results that are available in the existing literature are very limited in scope.The goal of this project is to take an important step to provide a systematic framework for understanding the conditions on the step size in numerical di^erentiation that are needed to obtain the optimal quality of approximation. These conditions involve subtle tradeoffs between the complexity of the function that needs to be differentiated and the amount of information that is available in the sample of data, and the degree of smoothness of the expectation of the function with respect to the sampling distribution. Empirical process theory provides a powerful tool for analyzing the complex of functions in the presence of randomly sampled data.This project focuses on analyzing the use of numerical derivatives in estimating the asymptotic variance of estimators and in obtaining extreme estimators through gradient based optimization routines. The PIs' first goal is to give general sufficient consistency conditions that allow for nondifferentiable and discontinuous moment functions in consistent variance estimation. The precise rate conditions for the step size in numerical differentiation that we obtain depend on the tradeoff between bias and the degree of nonsmoothness of the moment condition. These general conditionscan be specialized for certain continuous models, for which choosing a smaller step size can only be beneficial in reducing the asymptotic bias. However, the asymptotic bias will be dominated by the statistical noise once it falls below a certain threshold. The second goal of this project is to analyze a class of estimators that are based on numerically differentiating a finite sample objective function, and provide conditions under which numerical derivative based optimization methods deliver consistent and asymptotic normal parameter estimates. The conditions for numerical extreme estimators require that the step size used in thenumerical derivative has to converge to zero at specific rates when the sample size increases to infinity. The conditions required for the consistency of the asymptotic variance and for the convergence of the estimator itself can be different. The PIs seek extensive results that cover finite dimensionalparametric models, infinite dimensional semiparametric models, and models that are defined by U-processes involving multiple layers of summation over the sampling data. The proposed project involves joint work with Professor Aprajit Mahajan from Stanford University.