Issues in Maintaining Scientific Integrity in Applications of Automated Methodology

该项目由科学和社会计划中的伦理和价值观在科学、工程和社会中的组成部分资助，研究因失败而导致的科学完整性缺失，以比较首选分析方法和替代可用方法的准确性。当代应用科学是由其测量能力和伴随而来的计算能力的戏剧性增长所推动的，导致了从传统的假设和检验方法到搜索数据库以寻找假设的方法的根本变化。由于对成本、算法文档和数据访问的实际考虑，基于大型数据库的算法数据分析的工作结果通常不可能被独立的科学家小组复制或测试。因此，科学可靠性的堡垒之一--独立确认或反驳--即使没有失去，也会越来越少。因此，对于自动化或半自动科学的准确性和可靠性来说，重要的是制定专业标准，并在需要客观的经验测试、在明确假设下证明正确性、在具体应用中理解这些假设以及考虑解决类似目标的替代方法的各种科学中灌输专业标准。本项目将描述系统和仔细地对自动化和半自动程序进行验证的情况，并将它们与忽视评估算法准确性的一个或另一个重要方面的情况进行对比。案例将包括沿海海底测绘、基因调控网络的估计、矿物成分的机器人测定、使用因子分析和回归估计智力对行为的影响、估计低水平铅暴露对儿童智力的影响、野火预报以及肺炎预后的估计。调查将阐明关于验证搜索和预测程序的一般标准，并考虑有时不符合这些标准的各种原因，无论是在特定的例子中还是在更一般的情况下。这项工作将产生一本书，一个广泛使用的因果和统计推理自动化课程的模块，以及政策论文。这项研究的更广泛影响应该是提高对在应用科学中部署自动搜索程序的方法学问题的认识，并希望最终强调在许多领域指导学生建立搜索和预测程序准确性的条件的必要性，以及使用现有的最可靠、可行的程序的义务。

This project funded by the Ethics and Values in Science, Engineering and Society component of the Science and Society Program studies lapses of scientific integrity arising from failures to compare the accuracy of a preferred analysis method with that of alternative available methods. Contemporary applied science is driven by dramatic increases in both its measurement capacity and concomitant computational power, resulting in a fundamental change from a traditional hypothesize-and-test methodology to a search-databases-for-hypotheses methodology. The results of work with algorithmic data analysis, based on large databases are often essentially impossible for independent groups of scientists to replicate or test by virtue of practical considerations of cost, algorithm documentation, and data access. One of the bastions of scientific reliability, independent confirmation or refutation, is therefore increasingly mitigated if not lost Accordingly, it is important to the accuracy and reliability of automated or semi-automated science that professional standards develop and be inculcated in a wide range of sciences requiring objective, empirical testing, proofs of correctness under explicit assumptions, understanding of those assumptions in concrete applications, and consideration of alternative methods addressing similar goals. This project will describe cases in which validation of automated and semi-automated procedures has been done systematically and carefully, and contrast them with cases in which one or another important aspect of evaluation of the accuracy of algorithms has been neglected. Cases will include work on coastal sea floor mapping, estimation of gene regulation networks, robotic determination of mineral composition, uses of factor analysis and regression in estimating effects of intelligence on behavior, estimates of the effects of low level lead exposure on the intelligence of children, forecasts of wildfire, and estimates of pneumonia prognosis. The investigation will articulate general standards about validation of search and forecasting procedures, and consider a variety of reasons why such standards are sometimes not met, both in the particular examples and more generally. This work will result in a book, in a module for a widely used automated course on causal and statistical reasoning, and in policy essays. The broader impact in this research should be a heightened awareness of methodological issues in the deployment of automated search procedures in applied sciences, and, one hopes, eventually an emphasis in the instruction of students in many fields on the necessity of establishing the conditions for accuracy of search and forecasting procedures, and the obligation to use the most reliable, feasible procedures available.