Diagnostics for Structured Data and Quality Improvement

结构化数据诊断和质量改进

• 批准号: 9803622
• 负责人: Douglas Hawkins
• 金额: $ 9.21万
• 依托单位: University of Minnesota-Twin Cities
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 1998
• 资助国家: 美国
• 起止时间: 1998-08-15 至 2002-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=9803622&HistoricalAwards=false
• 关键词: Diagnostics; Structured; Data; Quality; Improvement

9803622Douglas M. HawkinsStructured data sets are those in which the data are other than independent identically distributed scalar quantities. Examples are multiple regression and multivariate data sets, and time series. Diagnostics involves identifying cases that depart from some baseline (usually Gaussian) model. This general framework covers finding outliers as a part of data analysis, and is also the basis for statistical process control (SPC) methodologies. One thread of the present project extends the PI's previous work in outlier identification, particularly in situations where the outliers are numerous or badly placed. It has recently become apparent that methods that work well on text-book-sized problems are useless in data sets with even a few thousand cases in a few dozen dimensions. As such data sets are increasingly common, this has reopened a major emphasis of the present project -- the whole question of workable approaches for finding outlying cases in large data sets. A somewhat distinct problem is detection of persistent changes in time-ordered scalar and multivariate data. This is the problem addressed by change-point, exponentially weighted moving average, and cumulative sum methodologies. The program of work includes a major effort in this area also. The union of the two problem areas leads to the design of statistical process control methodologies that are resistant to isolated outliers.In large data bases it is impossible to verify the correctness or internal consistency of the entries using current methodology. Methods that work well on small data sets are computationally unthinkable in data sets up in the megabyte and beyond range leaving the quality of information in data bases hostage to undetected errors. This project is developing methods to identify "outliers" --- atypical entries in large data bases --- with an acceptable, though still large, amount of computational effort. More processor power alone will not solve the problem, but more powerful processors combined with the improved algorithms developed in this program of work may do so. The problem is inherently amenable to distributed processing --- previous work showed how outlier identification could be speeded up by using an array of central processors. Another thread of the work is cumulative sum (cusum) charting, a tool in the statistical process control (SPC) family. The classic Shewhart Xbar and R control charts are incapable of detecting small but persistent shifts. Such shifts are found and diagnosed rapidly with cusums. Used in conjunction with Shewhart charts, cusums can diagnose manufacturing problems, leading to substantial quality improvement. Cusums are also effective in many other monitoring situations, from online medical monitoring to detecting plumes of pollution in air or water. Groundwater monitoring around a landfill, for example, aims at exactly this problem of detecting an increased level of pollutants against a highly variable background. Cusums are already recognized as a powerful tool to use in the detection and diagnosis of leakages, and their extension to handle non-detect chemical data is important in extending their applicability to pollutants like heavy metals that are harmful at low concentrations.

9803622道格拉斯M.结构化数据集是指数据不是独立同分布的标量的数据集。 例子是多元回归和多变量数据集，以及时间序列。 诊断涉及识别偏离某些基线（通常是高斯）模型的情况。 这个一般框架涵盖了作为数据分析一部分的异常值，也是统计过程控制（SPC）方法的基础。 本项目的一个线程扩展了PI以前在离群值识别方面的工作，特别是在离群值众多或位置不佳的情况下。 最近很明显，在教科书大小的问题上工作良好的方法在几十个维度的几千个案例的数据集上是无用的。 由于这类数据集越来越普遍，这就重新提出了本项目的一个主要重点-在大型数据集中寻找外围病例的可行方法的整个问题。 一个有点明显的问题是检测时间排序的标量和多变量数据中的持续变化。 这就是变点、指数加权移动平均和累积和方法所解决的问题。 工作方案也包括在这一领域作出重大努力。 这两个问题领域的结合导致了统计过程控制方法的设计，这些方法对孤立的离群值具有抵抗力。在大型数据库中，使用当前的方法不可能验证条目的正确性或内部一致性。 在小数据集上工作良好的方法在兆字节和超出范围的数据集上是计算上不可想象的，从而使数据库中的信息质量受到未检测到的错误的影响。 这个项目正在开发方法来识别“离群值”-大型数据库中的非典型条目-尽管计算工作量仍然很大，但可以接受。 更强大的处理器能力本身并不能解决这个问题，但更强大的处理器结合本工作计划中开发的改进算法可能会解决这个问题。 这个问题本质上是服从分布式处理-以前的工作表明如何离群识别可以通过使用中央处理器阵列来加速。 工作的另一个线程是累积和（cumulative sum）图表，统计过程控制（SPC）家族中的一种工具。 经典的ShewhartXbar和R控制图无法检测微小但持续的变化。 这种变化被发现和诊断迅速与累积。 与休哈特图结合使用，累积和可以诊断制造问题，从而导致实质性的质量改进。 累积求和在许多其他监测情况下也很有效，从在线医疗监测到检测空气或水中的污染羽流。 例如，对垃圾填埋场周围的地下水进行监测，就是为了解决这个问题，即在高度可变的背景下检测污染物水平的增加。 累积量已经被公认为是检测和诊断泄漏的有力工具，其扩展到处理非检测化学数据对于将其适用性扩展到低浓度有害的重金属等污染物非常重要。