Identification of animal behavioral strategies by inverse reinforcement learning.
Identification of animal behavioral strategies by inverse reinforcement learning.
复制标题
DOI：
10.1371/journal.pcbi.1006122
复制
发表时间：
2018-05
影响因子：
4.3
通讯作者：
Ishii S
中科院分区：
文献类型：
--
作者：
Yamaguchi S;Naoki H;Ikeda M;Tsukada Y;Nakano S;Mori I;Ishii S
动物能够通过控制各种行为模式在环境中达到期望的状态。识别用于这种控制的行为策略对于理解动物的决策非常重要，并且是解剖神经系统完成的信息处理的基础。然而，量化这种行为策略的方法尚未完全建立。在这项研究中，我们开发了一个逆学习（IRL）的框架，以确定动物的行为策略，从行为的时间序列数据。我们将这个框架应用于C。线虫趋温行为;在恒温培养后，有或无食物，喂养的蠕虫喜欢，而饥饿的蠕虫避免培养温度上的热梯度。我们的IRL方法显示，喂养的蠕虫使用的绝对温度和时间的衍生物，他们的行为涉及两种策略：定向迁移（DM）和等温迁移（IM）。有了DM，蠕虫有效地达到了特定的温度，这解释了它们在进食时的趋热行为。与IM，蠕虫移动沿着一个恒定的温度，这反映了等温跟踪，以及观察到在以前的研究。与喂养的动物相反，饥饿的蠕虫仅使用绝对温度而不是温度的时间导数来逃避培养温度。我们还研究了这些策略的神经基础，通过将我们的方法应用于热敏神经元缺陷的蠕虫。因此，我们的IRL为基础的方法是有用的，从行为时间序列数据中识别动物的策略，并可应用于广泛的行为研究，包括决策，在其他生物体。理解动物的决策过程一直是神经科学和行为生态学中的一个基本问题。许多研究分析了行为任务中代表决策的行动，其中奖励是人为设计的特定目标。然而，不可能将这种人工设计的实验扩展到自然环境中，因为在后者中，对自由行为的动物的奖励无法明确定义。为此，我们试图扭转当前的范式，以便可以从行为数据中识别奖励。在这里，我们提出了一种新的逆向工程方法（逆向强化学习），它可以从自由行为动物的时间序列数据中估计行为策略。将该技术应用于C.线虫的趋热性，我们成功地确定了各自的奖励为基础的行为策略。
Animals are able to reach a desired state in an environment by controlling various behavioral patterns. Identification of the behavioral strategy used for this control is important for understanding animals’ decision-making and is fundamental to dissect information processing done by the nervous system. However, methods for quantifying such behavioral strategies have not been fully established. In this study, we developed an inverse reinforcement-learning (IRL) framework to identify an animal’s behavioral strategy from behavioral time-series data. We applied this framework to C. elegans thermotactic behavior; after cultivation at a constant temperature with or without food, fed worms prefer, while starved worms avoid the cultivation temperature on a thermal gradient. Our IRL approach revealed that the fed worms used both the absolute temperature and its temporal derivative and that their behavior involved two strategies: directed migration (DM) and isothermal migration (IM). With DM, worms efficiently reached specific temperatures, which explains their thermotactic behavior when fed. With IM, worms moved along a constant temperature, which reflects isothermal tracking, well-observed in previous studies. In contrast to fed animals, starved worms escaped the cultivation temperature using only the absolute, but not the temporal derivative of temperature. We also investigated the neural basis underlying these strategies, by applying our method to thermosensory neuron-deficient worms. Thus, our IRL-based approach is useful in identifying animal strategies from behavioral time-series data and could be applied to a wide range of behavioral studies, including decision-making, in other organisms. Understanding animal decision-making has been a fundamental problem in neuroscience and behavioral ecology. Many studies have analyzed the actions representing decision-making in behavioral tasks, in which rewards are artificially designed with specific objectives. However, it is impossible to extend this artificially designed experiment to a natural environment, as in the latter, the rewards for freely-behaving animals cannot be clearly defined. To this end, we sought to reverse the current paradigm so that rewards could be identified from behavioral data. Here, we propose a new reverse-engineering approach (inverse reinforcement learning), which can estimate a behavioral strategy from time-series data of freely-behaving animals. By applying this technique on C. elegans thermotaxis, we successfully identified the respective reward-based behavioral strategy.
登录
查看更多内容
DOI：
10.1073/pnas.0610877104
发表时间：
2007-02-13
影响因子：
11.1
作者：
Chelur, Dattananda S.;Chalfie, Martin
通讯作者：
Chalfie, Martin
影响因子：
2.9
作者：
IWASA, Y;HIGASHI, M;YAMAMURA, N
通讯作者：
YAMAMURA, N
影响因子：
56.9
作者：
Kuhara, Atsushi;Okumura, Masatoshi;Mori, Ikue
通讯作者：
Mori, Ikue
影响因子：
1.9
作者：
Muelling, Katharina;Boularias, Abdeslam;Peters, Jan
通讯作者：
Peters, Jan
DOI：
10.1073/pnas.0805004105
发表时间：
2008-08-05
影响因子：
11.1
作者：
Biron, David;Wasserman, Sara;Sengupta, Piali
通讯作者：
Sengupta, Piali