SHF:Small:A Domain-Specific Language for Designing Cognitive-Science Experiments

当认知科学家设计一个实验时，设计通常会指定感兴趣的因素，多种因素的组合方式，以及避免无趣组合或平衡组合频率的约束。计算机使更复杂的实验设计成为可能，因为一个程序可以生成实验的步骤，但编写一个正确处理实验约束的程序可能很困难。更糟糕的是，程序中的细微错误可能会使生成的刺激产生偏差，从而使实验结果无效。该项目的新颖之处在于一种编程语言，它可以让科学家用更高层次的术语描述实验的设计——也就是说，比通用语言的循环和条件语句更高层次——以及一个执行引擎，从它的描述中生成实验的刺激。该项目的影响是使实验设计更快，为科学家提供更好的方式来交流他们的实验设计，并通过确保实验刺激的统计特性来提高实验的可重复性。生成的刺激具有良好统计特性的关键是对实验约束的解空间进行均匀采样。在一个简单的实验中，所有因素交叉，组合的权重相等，随机选择是无偏的，很容易实现。当对组合施加约束时，编程问题就会变得相当困难，例如在一行中有两倍的一种刺激，而在一行中有另一种刺激。最先进的工具，如UNIGEN，可以从一个有限的空间中取样，但这个空间必须用SAT公式来表达，这与认知科学家思考他们的设计的方式相去甚远。该项目将把这种抽样技术与托管和编译领域特定语言的工具结合起来，允许实验人员在领域术语中工作，同时弥合实验人员的观点和实现机制之间的差距。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

When a cognitive scientist designs an experiment, the design typically specifies factors of interest, the way that multiple factors are combined, and constraints that avoid combinations that are not interesting or that balance the frequency of combinations. Computers have enabled more sophisticated experimental designs, since a program can generate the steps for an experiment, but writing a program that correctly handles an experiment's constraints can be difficult. Worse, subtle mistakes in the program can bias the generated stimuli and invalidate the experiment's results. The project's novelties are a programming language that lets a scientist describe an experiment's design in higher level terms --- that is, higher than a general-purpose language's loops and conditionals --- and an execution engine to generate an experiment's stimuli from its description. The project's impacts are to make experiment design faster, to provide a better way for scientists to communicate their experiment designs, and to improve the reproducibility of experiments by ensuring statistical properties of the experiment's stimuli.The key to good statistical properties for generated stimuli is to uniformly sample the space of solutions to the experiment's constraints. In a simple experiment where all factors are crossed and combinations are weighted equally, random selection is unbiased and easy to implement. The programming problem is considerably more difficult when constraints are imposed on the combinations, such as having twice as many of one kind of stimuli in a row, versus having some other kind of stimuli in a row. State-of-the-art tools, such as UNIGEN, can sample from a constrained space, but the space must be expressed as a SAT formula, which is far from the way that cognitive scientists think about their designs. The project will combine this sampling technology with tools for hosting and compiling domain-specific languages, allowing an experimenter to work in domain terms while bridging the gap between the experimenter's view and the implementation mechanics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.