Multi-armed Bandit Problems with Covariates

多臂强盗（MAB）是指一类序列决策问题，其中在每一步中，需要选择一个群体，从中产生随机奖励。我们的目标是最大化总累积奖励。除了少数例外，关于MAB的文献忽略了可用的协变量。在这个项目中，PI将在一般框架中研究MAB协变量，并为各种应用开发方法和理论。该项目将1）提供选择关键协变量的方法; 2）建立变量选择的一致性; 3）建立累积奖励分配规则的一致性; 4）推导出累积奖励相对于Oracle选择的收敛速度。此外，将利用平均回报函数和模型组合的非参数估计来实现更高的预期回报。在医学实践中，以前在临床试验中被证明在人群水平上是最好的治疗方法，而对新患者的遗传特征等个人特征考虑最少。如果实际可行，患者有充分的理由接受治疗，即考虑到患有相同疾病的患者的所有既往治疗的结局，因此根据遗传信息、临床评估和所有累积的试验/治疗结果选择最有希望的个体化治疗。本研究将利用协变量序贯分配的统计机制，建立个体化用药的统计框架和理论与方法。 除医学外，序贯分配在运筹学、工业工程、经济学等领域也有应用。由于现代技术的指数增长提供了获取和处理信息的便利，随着新的研究带来关键预测因子的有效利用，协变量序贯分配的应用将产生真实的影响，挽救生命，改善健康，促进业务，降低社会运营成本。

Multi-armed bandit (MAB) refers to a class of sequential decision making problems where in each step one needs to choose a population from which a random reward will be generated. The goal is to maximize the total accumulated reward. The literature on MAB, with few exceptions, ignores available covariates. In this project, the PI will study MAB with covariates in general frameworks and develop methodologies as well as theories for various applications. The project will 1) provide methods for selecting key covariates; 2) establish consistency in variable selection; 3) establish consistency of the allocation rule in terms of the accumulated reward; 4) derive the rate of convergence of the accumulated reward relative to the oracle choices. In addition, nonparametric estimation of the mean reward functions and model combinations will be utilized for achieving higher expected reward. Strategies that simultaneously achieve high expected reward and also provide sufficient information for identifying the best arm (with high probability) will be sought.In practice of medicine, treatments previously shown to be the best at population levels in clinical trials are given to new patients with minimal consideration of his/her own personal characteristics such as genetic profile. If practically feasible, there is every reason for a patient to be treated in a way that the outcomes of all previous treatments of patients with the same disease will have been taken into account and consequently the most promising individualized treatment is selected based on genetic information, clinical assessments, and all the accumulated trial/treatment results. The proposed research will set up statistical frameworks and build theories and methodologies for application of individualized medicine using the statistical machinery of sequential allocation with covariates. Besides medicine, sequential allocation has applications in operations research, industrial engineering, economics and other fields. Due to the ease of getting and processing information furnished by the exponential growth of modern technology, with new research to bring effective use of key predictors, applications of sequential allocation with covariates will make a real impact, saving lives, improving health, promoting business, and reducing operating cost for the society.