Application of Reinforcement Learning to the Flight Control of Unmanned Aerial Vehicles

点击翻译按钮获取中文摘要

Project description:Complex urban environments pose a significant challenge for the operation of Unmanned Autonomous Systems (UAS). To operate in such areas, vehicles require the ability to rapidly change direction, avoid obstacles, and land in confined areas. This is especially challenging for a fixed-wing platform, due to the minimum airspeed needed to prevent aircraft stall. Fixed-wing platforms offer a number of advantages of rotary-wing vehicles, such as increased flight endurance and range, and greater payload capacity. As such, there is significant research in improving the agility of fixed-wing platforms, to improve their ability to operate in complex environments. This research proposal aims to build upon previous research projects conducted by the University of Bristol Flight Lab [1] [2]. These projects used a variable-sweep, fixed-wing platform to perform a bio-inspired perched landing manoeuvre. This agile landing manoeuvre, taking advantage of dynamic stall, enabled to UAS to land safely on small landing site, with minimal aircraft velocity, without the need for a long landing strip or arresting equipment. In particular, such a manoeuvre is applicable to challenging operational environments, such as in a complex urban setting, or operating from the deck of a ship. Non-linear control strategies were evaluated to generate the necessary perching manoeuvres. The reinforcement learning process, using a Deep Q-Network (DQN), generated trajectories with the lowest cost function, and showed the ability to generate trajectories from a range of starting conditions.Research in the first year aimed to modernise the perching UAV learning process, integrating and evaluating state-of-the-art reinforcement learning algorithms and frameworks. Compared to the DQN algorithm used previously, modern algorithms, such as Proximal Policy Optimisation (PPO), demonstrate the ability to attain higher rewards, as well as improved stability and convergence during the learning process. [3] This research has also explored the use of continuous control outputs, to increase the granularity of actuator control available to the learning agent. The next stage of this research is transitioning to real-world flight testing of the perching manoeuvre using these improvements to the process. This project has also transitioned to using state-of-the-art frameworks, such as OpenAI's Gym toolkit, to modernise and modularise the learning architecture. This lays the foundation for simpler, faster implementation of alternative algorithms and scenarios moving forward. This research project will aim to build on previous projects of the research group, and incorporate state-of-the-art algorithms and techniques, to develop reinforcement learning-based flight controllers which can perform a number of agile flight manoeuvres. The current flight dynamics model of a model UAV will be improved and expanded, to improve accuracy when performing agile manoeuvres, and by incorporating the lateral degrees of freedom into the current longitudinal-only model. Methods to improve the accuracy of the trained model will be evaluated and implemented, such as incorporating flight data into the offline, simulated learning process, and conducting online learning on the real-world vehicle. A number of agile flight manoeuvres, applicable to the operating in complex environments, will be selected, tested and evaluated. Examples of candidate algorithms include rapid changes of direction, and minimum distance 180 turns, such that the vehicle can avoid obstacles and navigate cluttered environments. A key focus of this research will be generating trained controllers and the necessary software frameworks such that they can be tested and used on real-world platforms.