RII Track-4: A Reflective Learning and Association Control Framework based on Adaptive Dynamic Programming: Architecture and Applications in Robotics

RII Track-4：基于自适应动态规划的反思性学习和关联控制框架：机器人技术的架构和应用

• 批准号: 1833005
• 负责人: Zhen Ni
• 金额: $ 26.15万
• 依托单位: South Dakota State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2019-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1833005&HistoricalAwards=false
• 关键词: RII; Track; Reflective; Learning; Association

Nontechnical description: Data efficiency and learning speed are two of the major bottlenecks for applying biologically-inspired control methods in many domains. The project's goal is to address these fundamental challenges by introducing a new adaptive dynamic programming-based learning control framework and integrate it into space robot navigation and scouting applications such as the Mars Rover. The scientific contribution of this project will promote interdisciplinary research in computational intelligence, machine learning, control and robotics. In addition to space applications, the proposed structure can also be applied to robot-assisted pedestrian evacuation application and cyber-physical power systems and is expected to impact general systems beyond this project period. Due to geographic isolation, South Dakota doesn't have a National Aeronautics and Space Administration (NASA) research center, and research collaboration opportunities on space technology is very limited. This project will expand the principle investigator (PI)'s research capacity through an extended visit and collaboration with NASA Ames Research Center located in San Jose, CA, and transform the PI's career path from theoretical algorithm/architecture development towards a new direction in complex space applications. Meanwhile, the outcomes of this project align well with the South Dakota's and South Dakota State University's strategic plans. The collaboration fits well with NASA's mission to Mars and technology roadmaps.Technical description: The proposed project will fundamentally advance the learning and association of biologically-inspired control methods. Three major contributions to the scientific field are expected. First, a new experience network is proposed and systematically integrated into a model-free adaptive dynamic programming-based learning control framework. The PI will design an experience replay tuple (i.e., state-action-reward pair) based on backward temporal difference information from historical data. This design can avoid the model network/prediction noted in existing literature and significantly save computation resources. Second, instead of a uniform sampling method, the PI proposes a prioritized sampling method based on the Bellman's estimation error. This new method is expected to enhance the controller's reflective learning performance with useful long-short term memory. The stability and convergence properties will also be analyzed. Third, this project is closely tied with NASA on robot and optimal control for space program. This new learning control structure will be integrated for robot navigation, exploration and scouting in unknown spaces. The PI and the collaborator will use both a virtual reality platform and a real Rover facility to analyze the control performance of the proposed algorithm at NASA Ames. The PI's outreach and dissemination plans will cultivate the scientific curiosity of K-12 students and motivate their interest in STEM programs. Moreover, the integration of the project's cutting-edge research results into the PI's new courses will aid retention of current STEM students. Specific plans include a workshop for a local middle school, a distance course for demographically diverse institutions, and development of new courses.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

非技术描述：数据效率和学习速度是在许多领域应用生物学启发的控制方法的两个主要瓶颈。该项目的目标是通过引入新的基于自适应动态编程的学习控制框架并将其集成到太空机器人导航和侦察应用程序（例如火星漫游者）中，以应对这些基本挑战。该项目的科学贡献将促进计算智能，机器学习，控制和机器人技术方面的跨学科研究。除空间应用外，所提出的结构还可以应用于机器人辅助的行人疏散应用和网络物理功率系统，并有望影响该项目时期以外的一般系统。由于地理隔离，南达科他州没有国家航空航天局（NASA）研究中心，并且关于太空技术的研究合作机会非常有限。该项目将通过与位于加利福尼亚州圣何塞的NASA AMES研究中心进行扩展访问和合作，扩大主要研究者的研究能力，并将PI的职业道路从理论算法/建筑发展转变为复杂空间应用中的新方向。同时，该项目的结果与南达科他州和南达科他州立大学的战略计划非常吻合。该协作非常符合美国宇航局（NASA）对火星和技术路线图的使命。技术描述：拟议的项目将从根本上提高生物启发的控制方法的学习和协会。预计对科学领域的三个主要贡献。首先，提出了一个新的体验网络，并系统地集成了基于无模型的自适应动态编程控制框架。 PI将根据历史数据的向后时间差异信息设计一个经验重播元组（即状态行动奖励对）。该设计可以避免在现有文献中指出的模型网络/预测，并显着节省计算资源。其次，PI不是统一的采样方法，而是根据Bellman的估计误差提出了优先的采样方法。这种新方法有望通过有用的长期记忆来增强控制器的反思性学习性能。还将分析稳定性和收敛属性。第三，该项目与机器人的NASA紧密相关，并为太空程序提供最佳控制。这种新的学习控制结构将集成在未知空间中的机器人导航，探索和侦察。 PI和合作者将同时使用虚拟现实平台和真正的流动站设施来分析NASA AMES所提出算法的控制性能。 PI的宣传和传播计划将培养K-12学生的科学好奇心，并激发他们对STEM计划的兴趣。此外，该项目的尖端研究结果集成到PI的新课程中将有助于保留当前的STEM学生。具体计划包括针对当地中学的研讨会，人口统计学多样化的机构的距离课程以及新课程的发展。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子和更广泛影响的评估审查标准来通过评估来获得支持的。

项目成果

Optimization of Merging Pedestrian Flows Based on Adaptive Dynamic Programming

• DOI: 10.23919/acc.2019.8814597
• 发表时间: 2019-07
• 期刊: 2019 American Control Conference (ACC)
• 作者: Chao Jiang;Yi Guo;Z. Ni;Haibo He

A Case Study of Horizon Window in Receding Horizon Control for Renewable Energy Integration

• DOI: 10.1109/eit.2019.8833852
• 发表时间: 2019-05
• 期刊: 2019 IEEE International Conference on Electro Information Technology (EIT)
• 作者: Avijit Das;Z. Ni

A Comparative Study of Smart Grid Security Based on Unsupervised Learning and Load Ranking

• DOI: 10.1109/eit.2019.8834059
• 发表时间: 2019-05
• 期刊: 2019 IEEE International Conference on Electro Information Technology (EIT)
• 作者: S. Paul;Md. Rashedul Haq;Avijit Das;Z. Ni

Study of Learning of Power Grid Defense Strategy in Adversarial Stage Game

• DOI: 10.1109/eit.2019.8834202
• 发表时间: 2019-05
• 期刊: 2019 IEEE International Conference on Electro Information Technology (EIT)
• 作者: S. Paul;Z. Ni

Robot-Assisted Pedestrian Regulation Based on Deep Reinforcement Learning

• DOI: 10.1109/tcyb.2018.2878977
• 发表时间: 2020-04
• 期刊: IEEE Transactions on Cybernetics
• 影响因子: 11.8
• 作者: Zhiqiang Wan;Chao Jiang;M. Fahad;Z. Ni;Yi Guo;Haibo He