KDI: New Algorithms, Architectures and Science for Data Mining of Massive Astrophysics Sky Surveys

点击翻译按钮获取中文摘要

Moore9873442 There many massive databases in industry and science, andthis is particularly true for Astrophysics. There are many kindsof questions that physicists and other users wish to ask thedatabases in real time, e.g., 'find outliers'; 'find clusters';'find patterns'; 'classify the data records into N predeterminedclasses.' Wide-ranging statistics and machine learningalgorithms similarly need to query databases, sometimes millionsof times for a single inference. With millions or billions ofrecords (such as the new generation of astrophysics sky surveys)this can be intractable using current algorithms. This projectaims to make repeated statistical querying of huge datasetscomputationally feasible by transforming massive databases intocondensed representations that permit the rapid answering of suchquestions. To achieve these goals, the investigator and hiscolleagues explore ways in which tools from statistics (such asBayesian networks), databases (such as kd-trees/R-trees), andArtificial Intelligence (such as AD-trees and rule-finders) canhelp, how they scale up, and how they can be combined. The investigators intend to help automate the process ofscientific discovery for astrophysical data sources in whichthere is too much information for any unaided human to have achance of spotting patterns, regularities, or anomalies.Government and industry in the U.S. have invested heavily iningenious new ways to gather information in all branches ofscience and industry, from cell biology to the flows of capitalin international commerce. Scientists and analysts who haveworked so hard to gather magnitudes more data than they had tenyears ago are now faced with an equally daunting task: exploitingit fully. It is ironic that in fields such as astrophysics thereis now so much data that no human has enough time to even see atiny fraction of it. The job of discovering new relationships,anomalies, and even causation, must now be at least partly turnedover to computers. The investigators comprise a team ofstatisticians, computer scientists, and astronomers who have eachalready made progress in this direction. This team develops newalgorithms to squeeze as much information as possible fromtrillion-byte astrophysics databases such as the Sloane SkySurvey. They also make sure that the resulting technology isdeployed elsewhere in science and industry.