ITR/IM: Statistical Data Mining for Cosmology

科学家们现在面临着许多非常大的高质量数据集。这些数据的潜在科学效益被分析它们以回答问题和测试理论的费力过程所抵消。本计画将发展新的资料探勘演算法，以达到电脑辅助探勘的目标。实现这一目标的两个关键问题是计算效率和自主性。 如果科学家要把精力集中在理解上，答案必须在几分钟内而不是几天内得到，因此需要效率。从数据挖掘和统计的角度来看，自治都很重要。对关系、模型和参数的详细搜索太大，人类无法手动进行。新的统计方法必须能够自主快速地选择模型，测试它们的意义，并将结果报告给搜索算法，以寻找新的发现。目前正在建设的国家虚拟天文台（NVO）是未来科学的典范。NVO将把来自多个多波长天空调查的PB级数据汇集到一个存储库中。新的方法将在宇宙学领域实施，但它们将适用于所有其他科学。这个项目的成员是计算机科学家、物理学家和统计学家，他们有着密切合作的记录。 他们一起工作，产生了：新的算法理论，新的统计理论，并公开领域的软件包从理论，同时开发新的课件和培训学生。该提案涉及以下领域的研究和教育：非参数数据分析。非参数统计模型支持功能强大的分析技术，这些技术只需做出最少的假设，这对科学准确性至关重要。统计模型可以直接用于发现。将单个对象与模型进行比较以识别异常，将生成的数据与理论模型进行比较以反驳或确认假设。该项目将建立在过去成功获得数量级加速的基础上，例如基于期望最大化的聚类和n点相关性，以使新方法更快。自动模拟参数搜索。使用所有上述方法，将开发一个系统，从参数化模拟和一些观测数据开始。系统将搜索参数空间，使用非参数方法针对真实的数据测试所得到的模拟，以确定最佳设置。

Scientists are now confronted with many very large high-quality data sets. The potential scientific benefits of these data are offset by the laborious process of analyzing them to answer questions and test theories. This project will develop new data mining algorithms in pursuit of the goal of computer assisted discovery. Two key issues in achieving this are computational efficiency and autonomy. If scientists are to focus their energy on understanding, answers must arrive in minutes rather than days, hence the need for efficiency. Autonomy is important both from the data mining and the statistical perspective. Detailed searches for relationships, models, and parameters are too large for humans to undertake manually. New statistical methods will have to autonomously and quickly select models, test their significance, and report the results to search algorithms looking for new discoveries.The National Virtual Observatory (NVO) currently under construction is a model of the future of science. The NVO will assemble petabytes of data from many multi-wavelength sky surveys into a single repository. The new methods to be developed will be implemented in the domain of cosmology, but they will be applicable to all other sciences. The members of this project are computer scientists, physicists and statisticians who have a track record of collaborating closely. Working together they have produced: new algorithmic theory, new statistical theory, and publicly fielded software packages resulting from the theory, while developing new courseware and training students.This proposal involves research and education in the following areas:Nonparametric data analysis. Nonparametric statistical models enable powerful analysis techniques that make minimal assumptions, which is critical for scientific accuracy.Automated discovery. Statistical models can be used directly for discovery. Individual objects are compared to models to identify anomalies and data generated models are compared to theoretical models to refute or confirm hypotheses.Computational methods for fast analysis. The project will build on past successes of getting orders of magnitude speedups on operations such as Expectation Maximization based clustering and n-point correlations to make the new methods fast.Automated simulation parameter searching. Using all of the above methods, a system will be developed that starts with a parameterized simulation and some observational data. The system will search the space of parameters, testing the resulting simulation against the real data using nonparametric methods to determine the best settings.