Bayesian Optimization for Exploratory Experimentation in the Behavioral Sciences

该研究项目将为人类行为研究开发一种探索性实验方法，使认知科学家能够有效地确定最佳条件——那些导致最稳健的学习、最快的表现、最少的错误、最好的决策和选择的条件。即将开发的工具将使科学家能够回答由于需要大量数据收集工作而无法解决的问题。为了理解和预测人类行为，科学家们通常会进行对照实验，对一组精心挑选的实验条件进行比较。例如，在设计教学软件时，可能会对两种教学方法进行比较。一种技术获得可靠的更好结果的发现具有实践和理论意义。然而，这个结果并没有回答人们经常想问的问题：什么是最好的技术？即将开发的方法将允许科学家在只有少数参与者的情况下评估许多实验条件，而传统的对照实验只评估几个条件，每个条件有许多参与者。该项目的一个关键产品将是黑盒软件，认知科学各个学科的研究人员可以用它来对自己领域的问题进行探索性实验。还将进行实验研究，以证明该方法在以下领域的广度：概念获取、色彩美学、正式教学以及可用和引人入胜的软件设计。该项目将把贝叶斯优化方法扩展到人体实验研究。贝叶斯优化长期以来一直用于地质统计学领域，用于从昂贵的测量（例如钻井测试）中推断未观测到的属性（例如，地表以下的石油储量）。在当前的项目中，正在探索的“景观”是在可能的条件下定义的（例如，训练策略），未观察到的属性是人类观察者的内部认知状态，并且通过行为评估（例如，学习评估）获得测量结果。为了将贝叶斯优化方法应用于一系列人体实验研究，将开发多种行为反应测量的数学模型，包括选择、排名、评级、延迟和自由回忆。该方法的探索性要求启发式方法依次选择实验条件，以在给定先前观察的情况下获得最大信息量的数据。各种启发式将在行为研究的背景下进行评估。

This research project will develop an exploratory experimentation methodology for human behavioral research that will allow cognitive scientists to efficiently identify optimal conditions -- those leading to the most robust learning, the fastest performance, the fewest errors, the best decisions and choices. The tools to be developed will allow scientists to answer questions they cannot currently address due to the massive data collection effort required. To understand and predict human behavior, scientists typically perform controlled experiments that compare a small, carefully chosen set of experimental conditions. For example, in designing instructional software, a comparison might be made between two techniques for teaching students. The finding that one technique obtains reliably better outcomes has both practical and theoretical implications. However, this result does not answer the question one often wishes to ask: what is the very best possible technique? The methodology to be developed will allow scientists to evaluate many experimental conditions with only a few participants, in contrast to the traditional controlled experiment which evaluates only a few conditions each with many participants. A key product of the project will be black-box software that researchers in various disciplines of the cognitive sciences can use to apply exploratory experimentation to problems in their own field. Experimental studies also will be conducted to demonstrate the breadth of the approach in domains including: concept acquisition, color aesthetics, formal instruction, and the design of usable and engaging software.The project will extend Bayesian optimization methods to human experimental research. Bayesian optimization has long been used in the geostatistics community for inferring unobserved properties (e.g., oil reserves below the earth's surface) from costly measurements (e.g., drilling tests). In the current project, the "landscapes" being explored are defined over possible conditions (e.g., training strategies), the unobserved properties are internal cognitive states of the human observer, and the measurements are obtained via behavioral evaluations (e.g., assessments of learning). To apply Bayesian optimization methods to a range of human experimental research, mathematical models will be developed for multiple behavioral response measures, including choice, ranking, rating, latency, and free recall. The exploratory nature of the approach requires heuristics for sequentially selecting experimental conditions to obtain maximally informative data given prior observations. Various heuristics will be evaluated in the context of behavioral research.