The timing mega-study: comparing a range of experiment generators, both lab-based and online.
The timing mega-study: comparing a range of experiment generators, both lab-based and online.
复制标题
作者：
Bridges D;Pitiot A;MacAskill MR;Peirce JW
行为科学的许多研究人员依赖于研究软件，该软件以亚毫秒的精度呈现刺激并记录响应时间。有大量的软件包可以用来进行这些行为实验，并测量参与者的响应时间和表现。然而，关于它们在实践中实现了什么样的定时性能的信息很少。在这里，我们报告了一项广泛的研究，着眼于视觉和听觉刺激的时间和响应时间的精度和准确性，测量与黑盒工具包。我们比较了一系列流行的软件包：PsychoPy、E-Prime®、NBS Presentation®、Psychophysics、OpenSesame、Expyriment、Gorilla、jsPsych、Lab.js和Testable。在可能的情况下，这些软件包在Windows、macOS和Ubuntu上进行了测试，并在一系列浏览器中进行了在线研究，以尝试识别常见的性能模式。在基于实验室的实验中，Psychtoolbox、PsychoPy、Presentation和E-Prime提供了最佳时机，在视觉、音频和响应测量方面的平均精度均低于1毫秒。OpenSesame在整体上的精确度略低，但最明显的是在音频刺激和Expyriment中的精确度相当差。在所有操作系统中，Ubuntu的精度通常比Windows稍好，而macOS是最差的，至少在视觉刺激方面，对于所有软件包来说。在线研究没有提供与实验室系统相同的精确度，所有测量结果的变异性略高。也就是说，PsychoPy和Gorilla在几个浏览器/操作系统组合上的表现最好，达到了非常接近毫秒的精度。对于响应时间（使用高性能按钮框测量），大多数软件包在所有浏览器中的精度至少低于10 ms，PsychoPy的精度低于3.5 ms。操作系统/浏览器组合之间存在相当大的差异，特别是在视听同步方面，这是基于浏览器的实验中最不精确的方面。尽管如此，数据表明，在线方法可以适用于广泛的研究，并适当考虑导致差异的来源。来自超过110，000次试验的结果突出了即使在这些专用软件包中也可能出现的各种计时质量。我们强调的重要性，科学家自己的时间验证测量自己的刺激和计算机配置。
Many researchers in the behavioral sciences depend on research software that presents stimuli, and records response times, with sub-millisecond precision. There are a large number of software packages with which to conduct these behavioral experiments and measure response times and performance of participants. Very little information is available, however, on what timing performance they achieve in practice. Here we report a wide-ranging study looking at the precision and accuracy of visual and auditory stimulus timing and response times, measured with a Black Box Toolkit. We compared a range of popular packages: PsychoPy, E-Prime®, NBS Presentation®, Psychophysics Toolbox, OpenSesame, Expyriment, Gorilla, jsPsych, Lab.js and Testable. Where possible, the packages were tested on Windows, macOS, and Ubuntu, and in a range of browsers for the online studies, to try to identify common patterns in performance. Among the lab-based experiments, Psychtoolbox, PsychoPy, Presentation and E-Prime provided the best timing, all with mean precision under 1 millisecond across the visual, audio and response measures. OpenSesame had slightly less precision across the board, but most notably in audio stimuli and Expyriment had rather poor precision. Across operating systems, the pattern was that precision was generally very slightly better under Ubuntu than Windows, and that macOS was the worst, at least for visual stimuli, for all packages. Online studies did not deliver the same level of precision as lab-based systems, with slightly more variability in all measurements. That said, PsychoPy and Gorilla, broadly the best performers, were achieving very close to millisecond precision on several browser/operating system combinations. For response times (measured using a high-performance button box), most of the packages achieved precision at least under 10 ms in all browsers, with PsychoPy achieving a precision under 3.5 ms in all. There was considerable variability between OS/browser combinations, especially in audio-visual synchrony which is the least precise aspect of the browser-based experiments. Nonetheless, the data indicate that online methods can be suitable for a wide range of studies, with due thought about the sources of variability that result. The results, from over 110,000 trials, highlight the wide range of timing qualities that can occur even in these dedicated software packages for the task. We stress the importance of scientists making their own timing validation measurements for their own stimuli and computer configuration.
登录
查看更多内容
DOI：
10.3758/bf03195575
发表时间：
2004-05-01
期刊：
BEHAVIOR RESEARCH METHODS INSTRUMENTS & COMPUTERS
影响因子：
--
作者：
Plant, RR;Hammond, N;Turner, G
通讯作者：
Turner, G
DOI：
10.1111/j.2044-8317.1989.tb01111.x
发表时间：
1989-05-01
影响因子：
2.6
作者：
ULRICH, R;GIRAY, M
通讯作者：
GIRAY, M
影响因子：
4.1
作者：
Brand, Andrew;Bradley, Michael T.
通讯作者：
Bradley, Michael T.
影响因子：
3.7
作者：
Garaizar P;Vadillo MA
通讯作者：
Vadillo MA
影响因子：
3.7
作者：
Schubert TW;Murteira C;Collins EC;Lopes D
通讯作者：
Lopes D