CHS: Medium: Scaling Qualitative Inductive Analysis through Computational Methods

该项目的重点是在定性归纳方法（QIMs）的背景下整合人和计算，其中专家深入参与文本语料库，如开放式调查，转录采访或社交媒体内容的集合。这种参与可以产生见解，但专业知识和时间的限制使得这些方法难以扩展到大型数据集。机器学习和自然语言处理（NLP）等技术可以从文本数据中挖掘某些类型的模式，即使是大型人类团队也不可行，这可能会提供一种前进的方式;然而，机器在理解语言的细微差别时会犯错误，缺乏人类分析师的上下文和专业知识，并且可能无法检测到解决特定问题所需的有趣的小规模模式。该项目的目标是通过逆转传统模型来扩大QIM的使用，在传统模型中，人类被用来验证计算结果（“人在环”），而不是从人类的洞察力开始，可以通过计算模型，支持和分析建议（“计算机在环”）来放大。该团队将与心理健康、公共卫生、灾难应对、政策制定和慈善事业等领域的合作者合作，对他们的QIM实践和需求进行定性研究，然后开发和评估系统，以提高专家可以产生的见解的质量和规模。 该小组将制作这些工具的公开版本，并通过一个在线社区向所有领域感兴趣的研究人员传播。 项目活动还将为信息可视化、人机交互和应用机器学习课程提供信息，沿着旨在招募高中女生从事计算机职业的讲习班。 第一条线索是对定性工作流程进行深入分析，使用文献中现有的定性工作账户和参与者观察方法，至少有10个团队从一系列学科，领域和规模中接近QIM。通过对访谈、日志和工件的分析，该团队将生成对这些工作流程的丰富描述，这些描述将进一步理解QIM作为一种方法，并确定适合计算支持的开放问题。第二条线是开发QIM的计算模型，该模型与分析师对定性数据的过程和判断保持一致，使用来自研究团队及其合作伙伴的各种数据集。由于许多QIM方法将特定文本段落标记为与特定概念相关，因此将这些文本段落作为正例，将附近未标记的数据作为负例，可以将QIM建模为一系列二元分类问题。 这将允许团队使用由领域知识和第一个线程的见解指导的NLP方法来为基于机器学习的模型生成特征。第三个线程涉及通过一系列的文档级、文档级和主题级可视化将这些模型连接到分析师的流程，这些可视化利用模型的预测来建议与给定文档中的概念相关的其他段落，跨文档的模式的分层聚合以支持提取更高级别的主题沿着合并和划分概念的方法，统计分析主位在语料库中的出现率。算法和工具将通过一系列针对现有分析数据集的离线测试进行评估，在研讨会设置中进行简短的分析挑战竞赛，以评估工具的可用性和反应，与合作伙伴进行为期三个月的纵向部署，包括每周半结构化问卷调查，了解其在实践中的可用性，并在一个在线社区中为系统提供支持和收集反馈，同时围绕这种风格的大数据分析建立一个实践方法社区。该奖项反映了NSF的法定基金会的使命是履行其使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评价，被认为值得支持。

This project focuses on the integration of people and computation in the context of qualitative inductive methods (QIMs), in which experts deeply engage with text corpora such as open-ended surveys, transcribed interviews, or collections of social media content. This engagement can produce insights, but constraints on expertise and time make these methods hard to scale to large datasets. Technologies like machine learning and natural language processing (NLP), which can mine certain kinds of patterns from text data at scales not feasible for even large teams of humans, may offer a way forward; however, machines make mistakes understanding the nuances of language, lack the context and expertise of human analysts, and may fail to detect interesting small-scale patterns necessary to solve particular problems. The goal of this project is to scale up the use of QIMs by inverting traditional models where humans are used to verify computational results ("human-in-the-loop"), starting instead with human insights that can be amplified through computational models, support, and suggestions for analysis ("computer-in-the-loop"). Working with collaborators in domains including mental health, public health, disaster response, policy making, and philanthropy, the team will conduct qualitative studies of their QIM practices and needs, then develop and evaluate systems with the goal of improving both the quality and scale of the insights experts can generate. The team will produce publicly available versions of the tools and disseminate them through an online community for interested researchers from all fields. The project activities will also inform courses on information visualization, human computer interaction, and applied machine learning, along with workshops aimed at recruiting high school women to careers in computing.The work is organized around three main threads. The first thread is to conduct a deep analysis of qualitative work processes, using both existing accounts of qualitative work in the literature and participant observation methods with at least 10 teams who approach QIM from a range of disciplines, domains, and scales. Through analysis of interviews, logs, and artifacts, the team will generate rich descriptions of these work processes that will further both understanding of QIMs as a method and identify open problems amenable to computational support. The second thread is to develop computational models of QIMs that align with analysts' processes and judgments around qualitative data, using a variety of datasets from the research team and their partners. Because many QIM methods label specific text passages as relevant to a specific concept, taking those as positive examples and nearby, unlabeled data as negative examples may allow QIMs to be modeled as a series of binary classification problems. This will allow the team to use NLP methods guided by domain knowledge and insights from the first thread to generate features for machine learning-based models. The third thread involves connecting these models to analysts' processes through a series of passage-level, document-level, and theme-level visualizations that leverage the models' predictions to suggest other passages relevant to a concept in a given document, hierarchical aggregation of patterns across documents to support the extraction of higher-level themes along with ways to merge and divide concepts, and statistical analysis of the prevalence of themes in corpora-level analysis. The algorithms and tools will be evaluated through a series of offline tests against existing analyzed datasets, short analysis challenge contests in workshop settings to evaluate usability and reactions to the tool, longitudinal three-month deployments with partners that involve weekly semi-structured questionnaires about their usability in practice, and in an online community to both provide support for and collect feedback about the system while growing a methodological community of practice around this style of big data analysis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.