A deep learning and model predictive control framework to control quadrotors and agile robots

In recent years, computer scientists have developed increasingly advanced algorithms for controlling the movements of robotic agents. These include model predictive control (MPC) techniques, which use a model of the agent’s dynamics to optimize its future behavior toward a given goal while simultaneously satisfying a number of constraints (e.g., cannot crash into obstacles).

Researchers at Technical University of Munich and University of Zürich recently created Real-time Neural MPC, a framework that integrates complex model architectures based on artificial neural networks (ANNs) in an MPC framework for agile robots (i.e., quadrotors—drones with four rotors). This concept, presented in IEEE Robotics and Automation Letters, builds on a previous approach developed by the University of Zurich’s Robotics and Perception Group.

“We came across the excellent work by the Robotics and Perception Group led by Davide Scaramuzza, and were immediately captivated to build upon their core idea of having data-driven (learned) components boosting ‘traditional’ control algorithms,” Tim Salzmann and Markus Ryll, researchers at the Autonomous Aerial Systems Group of Technical University of Munich, told Tech Xplore.

“After we worked out a proof-of-concept to generalize their approach, which uses Gaussian Processes (GPs), to generic Neural Networks (Deep Learning Models), we pitched our idea to the Robotics and Perception Group at the University of Zurich. From there on, technical work and experiments were jointly advanced between the two labs, sparking a new collaboration.”

The new framework proposed by Salzmann, Ryll and their colleagues combines deep learning models and online optimization of MPC. Expressive deep learning models are computationally heavy. However, by approximating them online in real-time the framework can leverage dedicated hardware (GPUs) to efficiently process these models, allowing their system to predict optimal actions for robots in real-time.

“The Real-time Neural MPC framework allows for the combination of two fields, optimal control, and deep learning while allowing for both parts to leverage their respective highly optimized frameworks and computational devices,” Salzmann and Ryll explained. “As such, we can perform deep learning computations in PyTorch/Tensorflow on a GPU while the control optimization is performed in compiled C code on a CPU. This allows using the power of deep learning in previously unachievable applications—for example in onboard optimal control of a Quadrotor.”

The researchers evaluate their framework in a series of experiments, both in a simulated and real-world environment. In these tests, they specifically use it to control the actions of a highly agile quadrotor in real-time.

Their results are very promising, as they can leverage the predictive power of neural network architectures with a parametric capacity more than 4,000 times larger than those previously used to control the movements of agile robots in real-time. They also find that compared to conventional MPC methods without a deep learning component, their framework can reduce positional tracking errors by up to 82%.

“In robotics, we seek expressive models of the dynamics of the controlled systems and their interaction with the environment (e.g., aerodynamic effects, tire friction, etc.),” Salzmann and Ryll said. “While these are often hard to formulate analytically, learning-based approaches, especially neural networks, can capture dynamics and interaction effects. However, the model accuracy scales with the size of the Neural Network. Real-time Neural MPC enables the use of deep learning models many magnitudes more powerful and performant compared to what was previously feasible in Model Predictive Control.”

As demonstrated by the recently released Nvidia Jetson platform, GPU chips are gradually finding their way into embedded systems. The framework created by this team of researchers could soon allow developers to leverage the high predictive power of advanced data-driven AI techniques to better model the dynamics of robots integrating GPU chips and their interactions with the environment, reducing the risk of accidents and improving their navigation capabilities.

“There are many open avenues for future research,” Salzmann and Ryll added. “One problem of deep learning approaches is that their output can be erratic for situations that are not part of the training data (Out Of Distribution OOD). Detecting these situations and providing a fallback for the control to stabilize the system would lead to robustness in OOD situations.”

© 2023 Science X Network