Bayesian Optimization for Exploratory Experimentation in the Behavioral Sciences

这项研究项目将开发一种用于人类行为研究的探索性实验方法，使认知科学家能够有效地识别最佳条件--那些导致最健壮的学习、最快的表现、最少的错误、最好的决定和选择的条件。即将开发的工具将允许科学家回答他们目前由于需要大量数据收集工作而无法解决的问题。为了理解和预测人类的行为，科学家通常会进行对照实验，比较一组精心选择的小实验条件。例如，在设计教学软件时，可以对两种教学技术进行比较。发现一种技术可以可靠地获得更好的结果，这一发现具有实际和理论意义。然而，这一结果并没有回答人们经常想要问的问题：什么是最好的可能技术？即将开发的方法将允许科学家只需几个参与者就能评估许多实验条件，而传统的对照实验只评估几个条件，每个实验有多个参与者。该项目的一个关键产品是黑盒软件，认知科学各学科的研究人员可以使用该软件对自己领域的问题进行探索性实验。该项目还将进行实验研究，以展示该方法在概念获取、色彩美学、形式教学以及可用和吸引人的软件设计等领域的广度。该项目将把贝叶斯优化方法扩展到人体实验研究。贝叶斯优化长期以来一直被用于地质统计学领域，用于从昂贵的测量(例如钻井测试)中推断未观察到的属性(例如，地球表面下的石油储量)。在目前的项目中，正在探索的“景观”是在可能的条件下定义的(例如，训练策略)，未观察到的属性是人类观察者的内部认知状态，测量是通过行为评估(例如，学习评估)获得的。为了将贝叶斯优化方法应用于一系列人类实验研究，将开发多种行为反应措施的数学模型，包括选择、排名、评级、延迟和自由回忆。该方法的探索性要求启发式方法顺序地选择实验条件，以获得给定先前观察的最大信息量的数据。各种启发式方法将在行为研究的背景下进行评估。

This research project will develop an exploratory experimentation methodology for human behavioral research that will allow cognitive scientists to efficiently identify optimal conditions -- those leading to the most robust learning, the fastest performance, the fewest errors, the best decisions and choices. The tools to be developed will allow scientists to answer questions they cannot currently address due to the massive data collection effort required. To understand and predict human behavior, scientists typically perform controlled experiments that compare a small, carefully chosen set of experimental conditions. For example, in designing instructional software, a comparison might be made between two techniques for teaching students. The finding that one technique obtains reliably better outcomes has both practical and theoretical implications. However, this result does not answer the question one often wishes to ask: what is the very best possible technique? The methodology to be developed will allow scientists to evaluate many experimental conditions with only a few participants, in contrast to the traditional controlled experiment which evaluates only a few conditions each with many participants. A key product of the project will be black-box software that researchers in various disciplines of the cognitive sciences can use to apply exploratory experimentation to problems in their own field. Experimental studies also will be conducted to demonstrate the breadth of the approach in domains including: concept acquisition, color aesthetics, formal instruction, and the design of usable and engaging software.The project will extend Bayesian optimization methods to human experimental research. Bayesian optimization has long been used in the geostatistics community for inferring unobserved properties (e.g., oil reserves below the earth's surface) from costly measurements (e.g., drilling tests). In the current project, the "landscapes" being explored are defined over possible conditions (e.g., training strategies), the unobserved properties are internal cognitive states of the human observer, and the measurements are obtained via behavioral evaluations (e.g., assessments of learning). To apply Bayesian optimization methods to a range of human experimental research, mathematical models will be developed for multiple behavioral response measures, including choice, ranking, rating, latency, and free recall. The exploratory nature of the approach requires heuristics for sequentially selecting experimental conditions to obtain maximally informative data given prior observations. Various heuristics will be evaluated in the context of behavioral research.