Issues in Maintaining Scientific Integrity in Applications of Automated Methodology

该项目由科学与社会计划的科学，工程和社会组成部分的伦理和价值观资助，研究由于未能将首选分析方法的准确性与替代方法进行比较而导致的科学诚信缺失。 当代应用科学是由其测量能力和随之而来的计算能力的显着增加，导致从传统的假设和测试方法到搜索数据库的假设方法的根本变化。基于大型数据库的算法数据分析的工作结果通常基本上不可能由独立的科学家小组复制或测试，因为实际考虑到成本，算法文档和数据访问。因此，对于自动化或半自动化科学的准确性和可靠性来说，重要的是专业标准的发展，并在广泛的科学中灌输，这些科学需要客观的、经验的测试、在明确的假设下的正确性证明、在具体应用中对这些假设的理解，并考虑实现类似目标的替代方法。这个项目将描述的情况下，自动化和半自动化程序的验证已经做了系统和仔细的，并将它们与案件中的一个或另一个重要方面的评估算法的准确性被忽视。案例将包括沿海海底测绘，基因调控网络的估计，矿物成分的机器人测定，使用因子分析和回归估计智力对行为的影响，估计低水平铅暴露对儿童智力的影响，野火的预测和肺炎预后的估计。调查将阐明关于搜索和预测程序验证的一般标准，并考虑在特定示例和更普遍的情况下有时不符合这些标准的各种原因。 这项工作将导致一本书，在一个广泛使用的因果和统计推理自动化课程的模块，并在政策论文。 这项研究的更广泛的影响应该是在应用科学中部署自动搜索程序的方法论问题的认识提高，人们希望，最终在许多领域的学生的教学中强调建立搜索和预测程序的准确性的必要性，以及使用最可靠的义务，可用的可行程序。

This project funded by the Ethics and Values in Science, Engineering and Society component of the Science and Society Program studies lapses of scientific integrity arising from failures to compare the accuracy of a preferred analysis method with that of alternative available methods. Contemporary applied science is driven by dramatic increases in both its measurement capacity and concomitant computational power, resulting in a fundamental change from a traditional hypothesize-and-test methodology to a search-databases-for-hypotheses methodology. The results of work with algorithmic data analysis, based on large databases are often essentially impossible for independent groups of scientists to replicate or test by virtue of practical considerations of cost, algorithm documentation, and data access. One of the bastions of scientific reliability, independent confirmation or refutation, is therefore increasingly mitigated if not lost Accordingly, it is important to the accuracy and reliability of automated or semi-automated science that professional standards develop and be inculcated in a wide range of sciences requiring objective, empirical testing, proofs of correctness under explicit assumptions, understanding of those assumptions in concrete applications, and consideration of alternative methods addressing similar goals. This project will describe cases in which validation of automated and semi-automated procedures has been done systematically and carefully, and contrast them with cases in which one or another important aspect of evaluation of the accuracy of algorithms has been neglected. Cases will include work on coastal sea floor mapping, estimation of gene regulation networks, robotic determination of mineral composition, uses of factor analysis and regression in estimating effects of intelligence on behavior, estimates of the effects of low level lead exposure on the intelligence of children, forecasts of wildfire, and estimates of pneumonia prognosis. The investigation will articulate general standards about validation of search and forecasting procedures, and consider a variety of reasons why such standards are sometimes not met, both in the particular examples and more generally. This work will result in a book, in a module for a widely used automated course on causal and statistical reasoning, and in policy essays. The broader impact in this research should be a heightened awareness of methodological issues in the deployment of automated search procedures in applied sciences, and, one hopes, eventually an emphasis in the instruction of students in many fields on the necessity of establishing the conditions for accuracy of search and forecasting procedures, and the obligation to use the most reliable, feasible procedures available.