KDI: New Algorithms, Architectures and Science for Data Mining of Massive Astrophysics Sky Surveys

摩尔9873442 在工业和科学领域有许多庞大的数据库，尤其是天体物理学。 物理学家和其他用户希望在真实的时间内向数据库提出许多问题，例如，'find outliers';'find clusters';'find patterns';'将数据记录分成N个预定的类。“广泛的统计和机器学习算法同样需要查询数据库，有时需要查询数百万次才能得出一个结论。 对于数百万或数十亿的记录（如新一代天体物理巡天），使用当前的算法可能难以处理。 该项目旨在通过将大量数据库转换为允许快速回答此类问题的压缩表示，使重复的统计查询在计算上可行。 为了实现这些目标，研究人员和他的同事们探索了统计工具（如贝叶斯网络），数据库（如kd树/R树）和人工智能（如AD树和规则发现者）可以帮助的方式，它们如何扩展，以及它们如何组合。 这些研究人员打算帮助天体物理学数据源的科学发现过程实现自动化，在这些数据源中有太多的信息，任何一个没有帮助的人都没有机会发现模式、缺陷或异常。美国政府和工业界已经投入大量资金，在从细胞生物学到国际商业中的资本流动的所有科学和工业分支中寻找新颖的方法来收集信息。 科学家和分析家们努力工作，收集比十年前多得多的数据，现在他们面临着同样艰巨的任务：充分利用这些数据。 具有讽刺意味的是，在天体物理学这样的领域，现在有如此多的数据，以至于没有人有足够的时间去看其中的一小部分。发现新的关系、异常甚至因果关系的工作现在必须至少部分地交给计算机。 调查人员包括一组统计学家、计算机科学家和天文学家，他们都已经在这个方向上取得了进展。 该团队开发了新的出租车，以尽可能多地从数万亿字节的天体物理学数据库（如斯隆巡天）中提取信息。 他们还确保所产生的技术被部署在科学和工业的其他领域。

Moore9873442 There many massive databases in industry and science, andthis is particularly true for Astrophysics. There are many kindsof questions that physicists and other users wish to ask thedatabases in real time, e.g., 'find outliers'; 'find clusters';'find patterns'; 'classify the data records into N predeterminedclasses.' Wide-ranging statistics and machine learningalgorithms similarly need to query databases, sometimes millionsof times for a single inference. With millions or billions ofrecords (such as the new generation of astrophysics sky surveys)this can be intractable using current algorithms. This projectaims to make repeated statistical querying of huge datasetscomputationally feasible by transforming massive databases intocondensed representations that permit the rapid answering of suchquestions. To achieve these goals, the investigator and hiscolleagues explore ways in which tools from statistics (such asBayesian networks), databases (such as kd-trees/R-trees), andArtificial Intelligence (such as AD-trees and rule-finders) canhelp, how they scale up, and how they can be combined. The investigators intend to help automate the process ofscientific discovery for astrophysical data sources in whichthere is too much information for any unaided human to have achance of spotting patterns, regularities, or anomalies.Government and industry in the U.S. have invested heavily iningenious new ways to gather information in all branches ofscience and industry, from cell biology to the flows of capitalin international commerce. Scientists and analysts who haveworked so hard to gather magnitudes more data than they had tenyears ago are now faced with an equally daunting task: exploitingit fully. It is ironic that in fields such as astrophysics thereis now so much data that no human has enough time to even see atiny fraction of it. The job of discovering new relationships,anomalies, and even causation, must now be at least partly turnedover to computers. The investigators comprise a team ofstatisticians, computer scientists, and astronomers who have eachalready made progress in this direction. This team develops newalgorithms to squeeze as much information as possible fromtrillion-byte astrophysics databases such as the Sloane SkySurvey. They also make sure that the resulting technology isdeployed elsewhere in science and industry.