KDI: New Algorithms, Architectures and Science for Data Mining of Massive Astrophysics Sky Surveys

工业和科学领域有许多庞大的数据库，天体物理学领域尤其如此。物理学家和其他用户希望实时向数据库询问许多类型的问题，例如，“找到异常值”；“找到集群”;“找到模式”;将数据记录分为N个预先确定的类。广泛的统计和机器学习算法同样需要查询数据库，有时为了一个推断需要查询数百万次。对于数百万或数十亿的记录（比如新一代的天体物理学天空调查），使用当前的算法可能很难做到这一点。该项目旨在通过将大量数据库转换为允许快速回答此类问题的浓缩表示，使大型数据集的重复统计查询在计算上可行。为了实现这些目标，研究者和他的同事们探索了统计学工具（如贝叶斯网络）、数据库（如kd树/ r树）和人工智能（如ad树和规则查找器）可以提供帮助的方法，以及它们如何扩大规模，以及如何将它们结合起来。研究人员打算帮助实现天体物理数据源的科学发现过程的自动化，因为其中的信息太多，任何没有帮助的人都没有机会发现模式、规律或异常。从细胞生物学到国际贸易中的资本流动，美国政府和工业界在科学和工业的各个部门都投入了大量的新颖的方法来收集信息。科学家和分析人士一直在努力收集比十年前多得多的数据，现在他们面临着同样艰巨的任务：充分利用这些数据。具有讽刺意味的是，在天体物理学等领域，现在有如此多的数据，以至于人类甚至没有足够的时间去看其中的一小部分。发现新的关系、异常甚至因果关系的工作，现在必须至少部分地交给计算机。研究人员组成了一个由统计学家、计算机科学家和天文学家组成的团队，他们已经在这个方向上取得了进展。这个团队开发了新的算法，从像斯隆巡天这样的万亿字节的天体物理数据库中尽可能多地提取信息。他们还确保由此产生的技术在其他科学和工业领域得到应用。

Moore9873442 There many massive databases in industry and science, andthis is particularly true for Astrophysics. There are many kindsof questions that physicists and other users wish to ask thedatabases in real time, e.g., 'find outliers'; 'find clusters';'find patterns'; 'classify the data records into N predeterminedclasses.' Wide-ranging statistics and machine learningalgorithms similarly need to query databases, sometimes millionsof times for a single inference. With millions or billions ofrecords (such as the new generation of astrophysics sky surveys)this can be intractable using current algorithms. This projectaims to make repeated statistical querying of huge datasetscomputationally feasible by transforming massive databases intocondensed representations that permit the rapid answering of suchquestions. To achieve these goals, the investigator and hiscolleagues explore ways in which tools from statistics (such asBayesian networks), databases (such as kd-trees/R-trees), andArtificial Intelligence (such as AD-trees and rule-finders) canhelp, how they scale up, and how they can be combined. The investigators intend to help automate the process ofscientific discovery for astrophysical data sources in whichthere is too much information for any unaided human to have achance of spotting patterns, regularities, or anomalies.Government and industry in the U.S. have invested heavily iningenious new ways to gather information in all branches ofscience and industry, from cell biology to the flows of capitalin international commerce. Scientists and analysts who haveworked so hard to gather magnitudes more data than they had tenyears ago are now faced with an equally daunting task: exploitingit fully. It is ironic that in fields such as astrophysics thereis now so much data that no human has enough time to even see atiny fraction of it. The job of discovering new relationships,anomalies, and even causation, must now be at least partly turnedover to computers. The investigators comprise a team ofstatisticians, computer scientists, and astronomers who have eachalready made progress in this direction. This team develops newalgorithms to squeeze as much information as possible fromtrillion-byte astrophysics databases such as the Sloane SkySurvey. They also make sure that the resulting technology isdeployed elsewhere in science and industry.