A flexible, web-based system for design, support, and delivery of adaptive tests

一个灵活的、基于网络的系统，用于设计、支持和交付自适应测试

• 批准号: 7746162
• 负责人: Robert John Wirth
• 金额: $ 8.24万
• 依托单位: VECTOR PSYCHOMETRIC GROUP LLC
• 依托单位国家: 美国
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-09-08 至 2010-08-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/7746162
• 关键词: Adopted; Algorithms; Arts; Attention; Award; Awareness; Behavior assessment; Behavioral; Behavioral Research; Behavioral Sciences; Clinical assessments; Communities; Computer software; Computers; Data Files; Development; Ensure; Equation; Feedback; Goals; Grant; Health Personnel; Improve Access; Individual; Information Systems; Internet; Knowledge; Length; Maintenance; Marketing; Measurement; Measures; Mental Health; Methodology; Modeling; National Institute of Mental Health; Neurosciences; Online Systems; Output; Participant; Patient Outcomes Assessments; Phase; Printing; Production; Psychometrics; Questionnaires; Research; Research Design; Research Personnel; Resources; Respondent; Scientist; Services; Site; Solutions; Statistical Computing; Support System; System; Systems Integration; Technology; Testing; Time; United States National Institutes of Health; Update; Validation; Writing; base; behavior measurement; computerized; computerized tools; design; experience; flexibility; improved; meetings; middleware; novel; programs; prototype; public health relevance; response; software development; theories; tool; trait; usability; user friendly software; user-friendly; web site

DESCRIPTION (provided by applicant): Awareness of the enormous advantages of computerized adaptive testing (CAT) in an item response theory framework (IRT) is growing rapidly in the behavioral research community. CAT has the capacity to significantly reduce respondent burden without losing measurement reliability. Although a small number of computerized adaptive tests (CATs) exist for behavioral measurement, they have been designed as "single-use" CATs which were developed to assess a fixed set of domains. The fact that currently available CATs cannot be easily altered by individuals other than the original developers to incorporate new domains severely limits the usability of the CATs beyond their initial purpose. Thus, despite the growing awareness of this state-of-the-art methodology, it is not currently feasible for the majority of behavioral researchers to implement CAT in their research designs. The construction and maintenance of CATs requires ongoing attention by individuals with extensive psychometric and computing experience. The few available software products for creating CATs are also extremely limited: they are PC-based so necessary software must be deployed on each computer that is going to be used for CAT administration; once software is installed on a PC that PC must be taken to the participants or the participants must be brought to the PC; subsequent software updates require internet connectivity or physical media; and these programs have extensive system requirements. The proposed project aims to overcome these barriers to CAT, making CAT an available tool for any behavioral researcher with access to the internet. [At the completion of Phase II] of the proposed project, a flexible web-based CAT system will be available so that any researcher can log-on to a web-site and create an adaptive test. [As a final goal,] the system will be flexible enough that users can, if desired, create and administer CATs remotely and without VPG assistance. However, researchers wishing to use this system will have the VPG staff available to them to help create and administer CATs, and to incorporate the resulting measures into their research design. The completion of Phase I will result in a prototype web-based CAT system. This [basic] prototype will be designed to receive input from a user through upload of files or direct user entry, incorporate the user input into a CAT algorithm implemented with IRT models, administer an adaptive test, and generate output for direct printing or download by the user. [The focus of Phase I is on the implementation of the core CAT algorithms and basic support input/output systems as dynamic web applications.]Phase II of the proposed project will solicit feedback from a wide array of potential users to expand the prototype, focusing on usability and continued flexibility, and will include pilot testing and refinement to ensure the web-based CAT system and support services are designed to fully meet the needs of behavioral research and assessment communities. PUBLIC HEALTH RELEVANCE: Computerized adaptive testing (CAT) has the capacity to significantly reduce respondent burden without any loss of reliability. However, CAT technology is currently accessible only to those with extensive psychometric knowledge thereby limiting the majority of behavioral researchers from implementing CAT in their own research. The proposed project aims to overcome these barriers to access by creating a flexible web-based CAT system that will allow behavioral researchers to incorporate this state-of-the-art methodology into their study designs.

描述(由申请人提供)：行为研究界对计算机化适应性测试(CAT)在项目反应理论框架(IRT)中的巨大优势的认识正在迅速增长。CAT有能力在不损失测量可靠性的情况下显著减轻受访者的负担。虽然有一小部分计算机化的适应性测试(CATS)用于行为测量，但它们被设计为“一次性使用”的CATS，其开发是为了评估一组固定的领域。除了最初的开发者之外，现有的CATS不能轻易地被个人更改以加入新的域，这一事实严重限制了CATS的可用性，使其超出了最初的目的。因此，尽管人们越来越意识到这一最先进的方法，但目前大多数行为研究人员在他们的研究设计中实施CAT是不可行的。CATS的构建和维护需要具有丰富心理测量和计算经验的个人持续关注。少数可用于创建CAT的软件产品也极其有限：它们是基于PC的，因此必须在用于CAT管理的每台计算机上部署必要的软件；一旦在PC上安装了软件，必须将PC带到参与者那里或必须将参与者带到PC上；后续软件更新需要互联网连接或物理介质；并且这些程序具有广泛的系统要求。拟议的项目旨在克服CAT的这些障碍，使CAT成为任何可以访问互联网的行为研究人员的工具。在拟议项目的[第二阶段]完成时，将提供一个灵活的基于网络的计算机辅助测试系统，以便任何研究人员都可以登录网站并创建自适应测试。[作为最终目标，]该系统将足够灵活，如果需要，用户可以在没有VPG帮助的情况下远程创建和管理CAT。然而，希望使用该系统的研究人员将让VPG工作人员帮助他们创建和管理CAT，并将由此产生的措施纳入他们的研究设计中。第一阶段的完成将产生一个基于网络的计算机辅助翻译系统的原型。这个[基本]原型将被设计为通过上传文件或直接用户输入来接收用户的输入，将用户输入合并到使用IRT模型实施的CAT算法中，管理自适应测试，并生成输出以供用户直接打印或下载。[第一阶段的重点是将核心计算机辅助测试算法和基本支持输入/输出系统作为动态网络应用程序加以实施。]拟议项目的第二阶段将征求广大潜在用户的反馈意见，以扩大原型，重点放在可用性和持续的灵活性上，并将包括试点测试和改进，以确保基于网络的计算机辅助测试系统和支持服务的设计充分满足行为研究和评估界的需要。公共卫生相关性： 计算机化自适应测验(CAT)能够在不损失任何可靠性的情况下显著减轻应答者的负担。然而，CAT技术目前仅对那些具有广泛心理测量学知识的人可用，从而限制了大多数行为研究人员在他们自己的研究中实施CAT。拟议的项目旨在通过创建一个灵活的基于网络的CAT系统来克服这些障碍，该系统将允许行为研究人员将这种最先进的方法纳入他们的研究设计中。