KDI: Learning Complex Motor Tasks in Natural and Artifical Systems

9873474Russell 该项目将开发一个统一的理论，说明自然和人工系统如何学习解决复杂的运动任务，例如跑步、潜水、投掷和飞行，这些任务需要大量的感官输入以及许多低级活动的协调、排序和微调。 这样的项目之所以成为可能，是因为我们对人类和其他动物运动控制系统的理解取得了重大的实验进展，并且因为我们的控制学习数学模型更加复杂。 这些模型不仅将用于分析和预测电机控制中的自然现象，而且还将用于为执行复杂任务的人工系统导出有效的自适应控制器。为了生成复杂的行为，自然和人工系统必须通过多层抽象进行分层组织。 因此，第一个研究任务将是确定可以对物理系统进行建模并可以定义控制操作的适当表示级别。 例如，在描述从 A 飞到 B 的昆虫时，可能的水平可能是 1) 控制每个翼猫详细形状的神经信号和机械特性 2) 基本翅膀拍动周期 3) 11 将周期转向直接飞行 4) 起飞、导航、着陆。将在各种实验环境和任务下进行详细的运动、力和/或气流测量，以建立正式模型和物理系统之间的对应关系。这些实验将针对各种生物体进行，可能包括飞行这些研究（对于昆虫，还有神经生理学研究）还将建立控制系统每个级别可用的感觉输入。考虑到控制系统的总体结构和适当的感觉输入，下一步是设计能够学习成功执行给定任务的学习算法。要使用的学习方法是强化学习，这是一种旨在调整控制算法以优化目标函数的技术。也就是说，奖励将作为感觉输入的一部分提供给学习算法，该算法将在分层控制结构中使用局部和全局奖励信号进行操作；此外，即使使用整体目标函数的非线性表示，这些算法也将被证明是收敛的，该研究应该阐明动物和人类的这种学习形式是否可以被视为由优化或其他原理驱动，例如保留各个级别之间的固定界面特征。在动物（尤其是人类）中发现一致的奖励功能将对一般学习理论产生重大影响。

9873474RussellThis project will develop a unified theory of how natural and artificial systems can learn to solve complex motor tasks, such as running, diving, throwing, and flying, that entail significant sensory input and the coordination., sequencing, and fine-tuning of many low-level activities. Such a project is possible because of significant experimental advances in our understanding of motor control systems in humans and other animals, and because of increased sophistication in our mathematical models of control learning. These models will be used not only to analyze and predict natural phenomena in motor control, but also to derive effective adaptive controllers for artificial systems carrying out complex tasks.To generate complex behaviors, natural and artificial systems must be organized hierarchically with multiple layers of abstraction. The first research task will therefore be to identify appropriate levels of representation at which the physical system can be modclled and at which control actions can be defined. For example, in describing an insect flying from A to B, possible levels might be 1) nerve signals and mechanical properties controlling the detailed shaping of each wingbcat 2) basic wingbeat cycle 3) 11 steering" the cycle to direct flight 4) takeoff, navigation, landing. Detailed motion, force, and/or airflow measurements will be made under a variety of experimental circumstances and tasks to establish the correspondence between formal models and physical systems. These experiments will be carried out for a variety of organisms, possibly including flying in insects, running in cockroaches, and for running, diving, and throwing in humans. These studies (and, in the case of insects, neurophysiological studies) will also establish the sensory inputs that are available at each level of the control system.Given the general structure of the control system and the appropriate sensory inputs, the next step is to design learning algorithms capable of learning to perform the given task successfully. The learning method to be used is reinforcement learning, a technique designed to adjust the control algorithm to optimize an objective function-that is, the long-term accumulated value of a specified reward signal. The reward is supplied to the learning algorithm as part of the sensory input. New reinforcement learning algorithms will be developed that operate using both local and global reward signals within a hierarchical control structure; furthermore, these algorithms will be proved to converge even using nonlinear representations of the overall objective function. This research should shed light on the central question of whether this form of learning in animals and humans can be viewed as driven by optimization or by some other principle, such as the preservation of fixed interface characteristics among the various levels of the system. Discovery of consistent reward functions in animals, especially humans, would have significant consequences for general theories of learning.