Nonhomogeneous Markov decision processes in medical decision making

这个建议的目的是开发一个数学框架，顺序随机决策时，一组可用的行动，其影响，随着时间的推移而变化。 建模为非齐次马尔可夫决策过程（MDP），这种情况适用于设备更换，设施位置和劳动力规划的决策。 我们的动机来自医疗决策问题。 例如，当每年数十亿美元投资于新疗法的研究和开发（R & D）时，患者应该何时开始或转换药物疗法？标准决策模型不考虑这种动态，但最近创建了网站，向医生及其患者传播正在进行的临床试验和已完成试验结果的信息。据推测，明确考虑未来有效疗法的不确定性的模型将产生与不考虑这种不确定性的模型不同的结果。 拟议的研究旨在量化不考虑不断变化的行动和影响的模型与考虑这些动态的模型之间的差异。 在此过程中，我们将开发用于估计描述研发的模型参数的方法，例如在未来不同时间可用的疗法的概率及其效果。 其他方法的贡献将包括发现最优解的结构特性和找到有效的求解算法。这项研究的总体贡献是在实际应用中的医疗决策问题，以及发现的基本见解时，考虑随时间变化的动作集在MDPs。

The purpose of this proposal is to develop a mathematical framework for sequential stochastic decision making when the set of available actions, and their effects, changes over time.  Modeled as nonhomogeneous Markov decision processes (MDPs), such situations apply to decisions about equipment replacement, facility location, and workforce planning.  Our motivation comes from medical decision making questions.  For example, when should a patient initiate or switch drug therapies when billions of dollars are invested annually in the research and development (R&D) of new therapies? Standard decision models do not consider this dynamic, yet websites have recently been created to disseminate information to physicians and their patients on clinical trials under way and results from trials already completed. Presumably, models that explicitly consider the uncertainty around effective therapies becoming available in the future would yield different results than models that do not consider this uncertainty.     The proposed research aims to quantify differences between models that do not consider evolving actions and effects, and models that do consider these dynamics.  In doing this, we will develop methods for estimating model parameters that describe R&D, such as probabilities of therapies becoming available at different times in the future, along with their effects.  Other methodological contributions will include discovering structural properties of optimal solutions and finding efficient solution algorithms. The overall contributions of this research are both in the practical applications to medical decision making problems as well as the discovery of fundamental insights when considering time-varying action sets in MDPs.