ITR/IM: Statistical Data Mining for Cosmology

科学家们现在面临着许多非常大的高质量数据集。为了回答问题和检验理论而对这些数据进行分析的费力过程，抵消了这些数据潜在的科学效益。该项目将开发新的数据挖掘算法，以实现计算机辅助发现的目标。实现这一目标的两个关键问题是计算效率和自主性。如果科学家要把精力集中在理解上，答案必须在几分钟而不是几天内得到，因此需要效率。从数据挖掘和统计的角度来看，自主性都很重要。对关系、模型和参数的详细搜索对于人工来说太大了，无法进行手动搜索。新的统计方法必须能够自主、快速地选择模型，测试它们的显著性，并将结果报告给搜索算法，以寻找新的发现。目前正在建设的国家虚拟天文台（NVO）是未来科学的典范。NVO将把来自许多多波长巡天的pb级数据整合到一个存储库中。将要开发的新方法将在宇宙学领域实施，但它们将适用于所有其他科学。这个项目的成员是计算机科学家、物理学家和统计学家，他们有着密切合作的记录。他们一起工作，产生了：新的算法理论，新的统计理论，以及由理论产生的公开的软件包，同时开发新的课件和培训学生。本建议涉及以下领域的研究和教育：非参数数据分析。非参数统计模型使强大的分析技术能够做出最小的假设，这对科学准确性至关重要。自动发现。统计模型可以直接用于发现。将单个对象与模型进行比较，以识别异常，将数据生成的模型与理论模型进行比较，以反驳或确认假设。快速分析的计算方法。该项目将以过去的成功为基础，在基于期望最大化的聚类和n点关联等操作上获得数量级的加速，以使新方法更快。自动仿真参数搜索。使用上述所有方法，将开发一个系统，该系统从参数化模拟和一些观测数据开始。系统将搜索参数空间，使用非参数方法对实际数据进行测试，以确定最佳设置。

Scientists are now confronted with many very large high-quality data sets. The potential scientific benefits of these data are offset by the laborious process of analyzing them to answer questions and test theories. This project will develop new data mining algorithms in pursuit of the goal of computer assisted discovery. Two key issues in achieving this are computational efficiency and autonomy. If scientists are to focus their energy on understanding, answers must arrive in minutes rather than days, hence the need for efficiency. Autonomy is important both from the data mining and the statistical perspective. Detailed searches for relationships, models, and parameters are too large for humans to undertake manually. New statistical methods will have to autonomously and quickly select models, test their significance, and report the results to search algorithms looking for new discoveries.The National Virtual Observatory (NVO) currently under construction is a model of the future of science. The NVO will assemble petabytes of data from many multi-wavelength sky surveys into a single repository. The new methods to be developed will be implemented in the domain of cosmology, but they will be applicable to all other sciences. The members of this project are computer scientists, physicists and statisticians who have a track record of collaborating closely. Working together they have produced: new algorithmic theory, new statistical theory, and publicly fielded software packages resulting from the theory, while developing new courseware and training students.This proposal involves research and education in the following areas:Nonparametric data analysis. Nonparametric statistical models enable powerful analysis techniques that make minimal assumptions, which is critical for scientific accuracy.Automated discovery. Statistical models can be used directly for discovery. Individual objects are compared to models to identify anomalies and data generated models are compared to theoretical models to refute or confirm hypotheses.Computational methods for fast analysis. The project will build on past successes of getting orders of magnitude speedups on operations such as Expectation Maximization based clustering and n-point correlations to make the new methods fast.Automated simulation parameter searching. Using all of the above methods, a system will be developed that starts with a parameterized simulation and some observational data. The system will search the space of parameters, testing the resulting simulation against the real data using nonparametric methods to determine the best settings.