Analyzing the SSA Disability Evaluation Process

点击翻译按钮获取中文摘要

Analytics (Objective 1) 1) Adjudicator Support Initial SSA decisions are made by examiners who have very little time to review evidence for a case, usually in the form of lengthy medical records from various healthcare providers. The objective of this focus area is to develop, adapt, and apply methods that aid SSA disability adjudicators to reach accurate, consistent, and timely decisions in accordance with SSA regulations and available medical evidence. This year, NIH deliverables encompassed the identification of functional terminology, specifically through a preliminary demonstration of named entity recognition and a demonstration of document classification. Identification of functional terminology: The aim of this subproject is to extract functional information from clinical documentation, which is an underdeveloped area of machine learning and natural language processing (NLP). In order to identify functional information, it is necessary to understand the variation in text used in clinical documentation, noting those rich in functional information compared to those focusing on health conditions. This year, we made progress on methods to characterize medical documentation with respect to functional language and extracting this information from samples of medical documents. To advance this work, we continued to develop annotation resources developed by content experts to serve as a gold standard for machine learning methods, models to automatically identify functional information, and resources to support the development and implementation of these models. We provided SSA with models for labeling and ranking medical records based on information related to mobility, self-care, and domestic life. During this fiscal year, we also drafted an ontology for mental functioning and generated a corresponding terminology. Automating the extraction, retrieval, and classification of functional information is intended to improve the efficiency of SSAs processes. WD-FAB development (Objective 2) In collaboration with the SSA, the NIH and Boston University developed a comprehensive and efficient assessment instrument called the Work Disability Functional Assessment Battery (WD-FAB). Contemporary models of disability indicate that in order to assess work disability, what individuals can do and what they are expected to do for work must both be assessed. The WD-FAB is intended to assess what individuals can do. The WD-FAB is a 15-20-minute individualized assessment of functional activity that uses Item Response Theory (IRT), along with computer adaptive technology (CAT), to select the most relevant test items from a large pool of items to measure self-reported functional ability. Item-based scoring means respondents do not need to answer all items or the same items to obtain comparative scores and scores are obtained in a highly efficient manner. This year, development of the WD-FAB has focused on implementing optimal methods in item response theory into the WD-FAB software. In addition, SSA commissioned the design of a preliminary pilot study examining the WD-FAB as applied to the continuing disability review (CDR) process to inform considerations for a future large scale study and potential implementation in the CDR process. 2) Functional Assessment Tools The objective of this focus area is to develop new ways to collect, structure, and interpret functional data for use by SSA. This work will include development of the WD-FAB and methods to assist in interpreting WD-FAB results. WD-FAB instrument development: The aim of this subproject is to finalize the development of the WD-FAB so that it is ready for real-world, applied testing. The instrument now includes over 300 items across eight domains, four of which represent physical function (basic mobility, upper body function, fine motor function, community mobility) and four of which represent mental health function (communication & cognition, resilience & sociability, self-regulation, and mood & emotions). Functional stages (e.g., low, moderate, high functioning) were developed by content experts to aid score interpretation. To date, the reliability and validity of the WD-FAB have been supported by a variety of evidence from a continuum of studies. This year, we provided SSA with iterations of the WD-FAB code to be used in future studies that look at whether the instrument is robust again intentional misreporting or whether additional features can be added to flag suspected instances of intentional misreporting. We are also working with collaborators at the University of New Hampshire to study how WD-FAB scores align with job demands. This work is ongoing. Continuing Disability Reviews and change in function over time: Once an individual is awarded disability benefits, their disability status is reassessed periodically. Following development of the WD-FAB, SSA requested creation of a preliminary pilot study, as well as a large scale study, to examine the utility of the WD-FAB in the SSA continuing disability review process. These study designs were delivered to SSA last year and SSA is in the process of securing a survey research firm for data collection. In preparation for analyzing these data, it is important to understand how function changes over time for individuals with disability. During this year, we have used SSA administrative data to examine how beneficiaries function changed over time. We will also be looking at WD-FAB data that has been collected over time as part of an SSA demonstration project to help understand this question as specifically relates to the WD-FAB. Publications generated by this year's research: Newman-Griffis D, Porcino J, Zirikly A, Thieu T, Camacho Maldonado J, Ho PS, Ding M, Chan L, Rasch E. Broadening horizons: the case for capturing function and the role of health informatics in its use. BMC Public Health. 2019 Oct 15;19(1):1288. doi: 10.1186/s12889-019-7630-3. Newman-Griffis D, Fosler-Lussier E. HARE: a Flexible Highlighting Annotator for Ranking and Exploration. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations. 2019 November; 85-90. doi: 10.18653/v1/D19-3015. Newman-Griffis D, Fosler-Lussier E. Writing habits and telltale neighbors: analyzing clinical concept usage patterns with sublanguage embeddings. Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019). 2019 November; doi: 10.18653/v1/D19-6218. Tang L. Group Sequential Comparison of Recall Curves. Presentation, The Second Symposium in Biostatistics and Data Science. 2019 November. Newman-Griffis D, Zirikly A, Ho PS, Camacho Maldonado J, Sacco M, Marr A, Lai AM, Fosler-Lussier E. Automated classification of mobility activities in free text clinical narratives. Poster presentation, 2019 Annual Symposium of the American Medical Informatics Association. Chang JC, Vattikuti S, Chow CC. Probabilistically-autoencoded horseshoe-disentangled multidomain item-response theory models. NeurIPS 2019 Bayesian Deep Learning Workshop. arXiv:1912.02351. Ye X, Tang LL, Zhu X. Group sequential comparison of positive predictive value curves for correlated biomarker data. Statistics in Medicine. 2020; 1 14. https://doi.org/10.1002/sim.8509. Chang J, Vattikuti S, Chow C. "Probabilistically-autoencoded horseshoe-disentangled multidomain item-response theory models." Bulletin of the American Physical Society. 2020.