A reinforcement learning model to inform optimal decision paths for HIV elimination.
A reinforcement learning model to inform optimal decision paths for HIV elimination.
复制标题
DOI：
10.3934/mbe.2021380
复制
发表时间：
2021-09-06
期刊：
影响因子：
--
通讯作者：
Gopalappa C
中科院分区：
文献类型：
--
作者：
Khatami SN;Gopalappa C
“结束艾滋病毒流行病（EHE）”国家计划旨在将美国的艾滋病毒年发病率从2015年的38，000减少到2025年的9，300和2030年的3，300。诊断和治疗是两种最有效的干预措施，因此，确定相应的检测和保留护理率的最佳组合将有助于为相关计划的实施提供信息。考虑到疾病的动态和随机复杂性以及决策的时间动态性，使用常用的参数优化方法或对预选选项进行详尽评估来求解最佳组合是不可行的。强化学习（RL）是一种理想的人工智能方法;然而，对于大规模随机问题，训练RL算法并确保收敛到最优值在计算上具有挑战性。我们在EHE目标的背景下评估其可行性。我们训练了一种RL算法，以确定2015-2070年期间每5年一次的艾滋病毒检测和保留护理率组合的“序列”，从而最佳地消除艾滋病毒。我们将最优性定义为最大化质量调整生命年并最小化HIV检测和护理和治疗成本的序列。我们表明，通过使用代理决策指标进行适当的重新制定来解决测试和保留率，克服了RL的计算挑战。我们使用了一个随机的基于代理的模拟来训练RL算法。由于有可变性的支持计划需要解决的障碍，以获得护理，我们评估了最佳决策的敏感性，三个成本函数。该模型建议扩大保留护理计划，以实现和保持高的年保留率，同时开始与高检测频率，但放宽它超过10年的时间，发病率下降。结果主要是稳健的成本的不确定性。然而，仅检测和保留护理并不能实现2030年EHE目标，这表明需要采取额外的干预措施。该模型的结果显示了收敛性。RL适用于评估传染病控制的阶段性公共卫生决策。
The ‘Ending the HIV Epidemic (EHE)’ national plan aims to reduce annual HIV incidence in the United States from 38,000 in 2015 to 9,300 by 2025 and 3,300 by 2030. Diagnosis and treatment are two most effective interventions, and thus, identifying corresponding optimal combinations of testing and retention-in-care rates would help inform implementation of relevant programs. Considering the dynamic and stochastic complexity of the disease and the time dynamics of decision-making, solving for optimal combinations using commonly used methods of parametric optimization or exhaustive evaluation of pre-selected options are infeasible. Reinforcement learning (RL), an artificial intelligence method, is ideal; however, training RL algorithms and ensuring convergence to optimality are computationally challenging for large-scale stochastic problems. We evaluate its feasibility in the context of the EHE goal. We trained an RL algorithm to identify a ‘sequence’ of combinations of HIV-testing and retention-in-care rates at 5-year intervals over 2015–2070 that optimally leads towards HIV elimination. We defined optimality as a sequence that maximizes quality-adjusted-life-years lived and minimizes HIV-testing and care-and-treatment costs. We show that solving for testing and retention-in-care rates through appropriate reformulation using proxy decision-metrics overcomes the computational challenges of RL. We used a stochastic agent-based simulation to train the RL algorithm. As there is variability in support-programs needed to address barriers to care-access, we evaluated the sensitivity of optimal decisions to three cost-functions. The model suggests to scale-up retention-in-care programs to achieve and maintain high annual retention-rates while initiating with a high testing-frequency but relaxing it over a 10-year period as incidence decreases. Results were mainly robust to the uncertainty in costs. However, testing and retention-in-care alone did not achieve the 2030 EHE targets, suggesting the need for additional interventions. The results from the model demonstrated convergence. RL is suitable for evaluating phased public health decisions for infectious disease control.
登录
查看更多内容
DOI：
10.1097/qai.0b013e31825bd862
发表时间：
2012-09-01
影响因子：
3.6
作者：
Gopalappa, Chaitra;Farnham, Paul G.;Sansom, Stephanie L.
通讯作者：
Sansom, Stephanie L.
DOI：
10.15585/mmwr.mm6647e1
发表时间：
2017-12-01
期刊：
MMWR. Morbidity and mortality weekly report
影响因子：
--
作者：
Dailey AF;Hoots BE;Hall HI;Song R;Hayes D;Fulton P Jr;Prejean J;Hernandez AL;Koenig LJ;Valleroy LA
通讯作者：
Valleroy LA
影响因子：
3.6
作者：
Brandeau, Margaret L.;Zaric, Gregory S.
通讯作者：
Zaric, Gregory S.
DOI：
10.1016/j.ssmph.2018.03.007
发表时间：
2018-04
期刊：
SSM - population health
影响因子：
--
作者：
Kreatsoulas C;Subramanian SV
通讯作者：
Subramanian SV
影响因子：
3.7
作者：
Lansky A;Johnson C;Oraka E;Sionean C;Joyce MP;DiNenno E;Crepaz N
通讯作者：
Crepaz N