RII Track-4: A Reflective Learning and Association Control Framework based on Adaptive Dynamic Programming: Architecture and Applications in Robotics

RII Track-4：基于自适应动态规划的反思性学习和关联控制框架：机器人技术的架构和应用

• 批准号: 1833005
• 负责人: Zhen Ni
• 金额: $ 26.15万
• 依托单位: South Dakota State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2019-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1833005&HistoricalAwards=false
• 关键词: RII; Track; Reflective; Learning; Association

Nontechnical description: Data efficiency and learning speed are two of the major bottlenecks for applying biologically-inspired control methods in many domains. The project's goal is to address these fundamental challenges by introducing a new adaptive dynamic programming-based learning control framework and integrate it into space robot navigation and scouting applications such as the Mars Rover. The scientific contribution of this project will promote interdisciplinary research in computational intelligence, machine learning, control and robotics. In addition to space applications, the proposed structure can also be applied to robot-assisted pedestrian evacuation application and cyber-physical power systems and is expected to impact general systems beyond this project period. Due to geographic isolation, South Dakota doesn't have a National Aeronautics and Space Administration (NASA) research center, and research collaboration opportunities on space technology is very limited. This project will expand the principle investigator (PI)'s research capacity through an extended visit and collaboration with NASA Ames Research Center located in San Jose, CA, and transform the PI's career path from theoretical algorithm/architecture development towards a new direction in complex space applications. Meanwhile, the outcomes of this project align well with the South Dakota's and South Dakota State University's strategic plans. The collaboration fits well with NASA's mission to Mars and technology roadmaps.Technical description: The proposed project will fundamentally advance the learning and association of biologically-inspired control methods. Three major contributions to the scientific field are expected. First, a new experience network is proposed and systematically integrated into a model-free adaptive dynamic programming-based learning control framework. The PI will design an experience replay tuple (i.e., state-action-reward pair) based on backward temporal difference information from historical data. This design can avoid the model network/prediction noted in existing literature and significantly save computation resources. Second, instead of a uniform sampling method, the PI proposes a prioritized sampling method based on the Bellman's estimation error. This new method is expected to enhance the controller's reflective learning performance with useful long-short term memory. The stability and convergence properties will also be analyzed. Third, this project is closely tied with NASA on robot and optimal control for space program. This new learning control structure will be integrated for robot navigation, exploration and scouting in unknown spaces. The PI and the collaborator will use both a virtual reality platform and a real Rover facility to analyze the control performance of the proposed algorithm at NASA Ames. The PI's outreach and dissemination plans will cultivate the scientific curiosity of K-12 students and motivate their interest in STEM programs. Moreover, the integration of the project's cutting-edge research results into the PI's new courses will aid retention of current STEM students. Specific plans include a workshop for a local middle school, a distance course for demographically diverse institutions, and development of new courses.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

非技术性描述：数据效率和学习速度是在许多领域应用生物启发控制方法的两个主要瓶颈。该项目的目标是通过引入一个新的基于自适应动态编程的学习控制框架来解决这些根本性挑战，并将其集成到火星漫游者等空间机器人导航和侦察应用中。该项目的科学贡献将促进计算智能，机器学习，控制和机器人技术的跨学科研究。除了空间应用外，拟议的结构还可以应用于机器人辅助行人疏散应用和网络物理电力系统，预计将在本项目期间对一般系统产生影响。由于地理位置偏僻，南达科他州没有美国国家航空航天局（NASA）的研究中心，空间技术的研究合作机会非常有限。该项目将通过与位于加利福尼亚州圣何塞的美国宇航局艾姆斯研究中心的长期访问和合作，扩大首席研究员（PI）的研究能力，并将PI的职业道路从理论算法/架构开发转向复杂空间应用的新方向。同时，该项目的成果与南达科他州和南达科他州州立大学的战略计划保持一致。该合作非常符合NASA的火星使命和技术路线图。技术描述：拟议的项目将从根本上推进生物启发控制方法的学习和关联。预计将对科学领域作出三大贡献。首先，提出了一种新的经验网络，并将其系统地集成到基于无模型自适应动态规划的学习控制框架中。PI将设计经验重放元组（即，状态-动作-回报对）。这种设计可以避免现有文献中提到的模型网络/预测，并显着节省计算资源。其次，PI提出了基于Bellman估计误差的优先采样方法，而不是均匀采样方法。这种新方法有望提高控制器的反射学习性能与有用的长短期记忆。并对算法的稳定性和收敛性进行了分析。第三，该项目与NASA在机器人和空间计划的最优控制方面密切相关。这种新的学习控制结构将被集成用于机器人在未知空间中的导航、探索和侦察。PI和合作者将使用虚拟现实平台和真实的漫游车设施来分析NASA艾姆斯提出的算法的控制性能。PI的推广和传播计划将培养K-12学生的科学好奇心，并激发他们对STEM项目的兴趣。此外，将该项目的前沿研究成果融入PI的新课程将有助于留住现有的STEM学生。具体计划包括为当地中学举办讲习班，为人口结构多样化的机构举办远程课程，以及开发新课程。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Optimization of Merging Pedestrian Flows Based on Adaptive Dynamic Programming

• DOI: 10.23919/acc.2019.8814597
• 发表时间: 2019-07
• 期刊: 2019 American Control Conference (ACC)
• 作者: Chao Jiang;Yi Guo;Z. Ni;Haibo He

Robot-Assisted Pedestrian Regulation Based on Deep Reinforcement Learning

• DOI: 10.1109/tcyb.2018.2878977
• 发表时间: 2020-04
• 期刊: IEEE Transactions on Cybernetics
• 影响因子: 11.8
• 作者: Zhiqiang Wan;Chao Jiang;M. Fahad;Z. Ni;Yi Guo;Haibo He

Study of Learning of Power Grid Defense Strategy in Adversarial Stage Game

• DOI: 10.1109/eit.2019.8834202
• 发表时间: 2019-05
• 期刊: 2019 IEEE International Conference on Electro Information Technology (EIT)
• 作者: S. Paul;Z. Ni

Pedestrian Flow Optimization to Reduce the Risk of Crowd Disasters Through Human–Robot Interaction

• DOI: 10.1109/tetci.2019.2930249
• 发表时间: 2020-06
• 期刊: IEEE Transactions on Emerging Topics in Computational Intelligence
• 影响因子: 5.3
• 作者: Chao Jiang;Z. Ni;Yi Guo;Haibo He

A Case Study of Horizon Window in Receding Horizon Control for Renewable Energy Integration

• DOI: 10.1109/eit.2019.8833852
• 发表时间: 2019-05
• 期刊: 2019 IEEE International Conference on Electro Information Technology (EIT)
• 作者: Avijit Das;Z. Ni