Deep deterministic policy gradient with constraints for gait optimisation of biped robots

Article type: Research Article

Authors: Liu, Xingyang^a | Rong, Haina^{a; *} | Neri, Ferrante^{b; e; *} | Yue, Peng^c | Zhang, Gexiang^d

Affiliations: [a] School of Electrical Engineering, Southwest Jiaotong University, Chengdu, Sichuan, China | [b] Nature Inspired Computing and Engineering Research Group, School of Computer Science and Electronic Engineering, University of Surrey, Guildford, UK | [c] State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, Shenyang, Liaoning, China | [d] School of Control Engineering, Chengdu University of Information Technology, Chengdu, Sichuan, China | [e] School of Software, Nanjing University of Information Science and Technology, Jiangsu, China

Correspondence: [*] Corresponding authors: Haina Rong, School of Electrical Engineering, Southwest Jiaotong University, Chengdu, Sichuan 610031, China. E-mail: [email protected]. Ferrante Neri, Nature Inspired Computing and Engineering Research Group, School of Computer Science and Electronic Engineering, University of Surrey, Guildford, Surrey GU2 7XH, UK. E-mail: [email protected].

Abstract: In this paper, we propose a novel Reinforcement Learning (RL) algorithm for robotic motion control, that is, a constrained Deep Deterministic Policy Gradient (DDPG) deviation learning strategy to assist biped robots in walking safely and accurately. The previous research on this topic highlighted the limitations in the controller’s ability to accurately track foot placement on discrete terrains and the lack of consideration for safety concerns. In this study, we address these challenges by focusing on ensuring the overall system’s safety. To begin with, we tackle the inverse kinematics problem by introducing constraints to the damping least squares method. This enhancement not only addresses singularity issues but also guarantees safe ranges for joint angles, thus ensuring the stability and reliability of the system. Based on this, we propose the adoption of the constrained DDPG method to correct controller deviations. In constrained DDPG, we incorporate a constraint layer into the Actor network, incorporating joint deviations as state inputs. By conducting offline training within the range of safe angles, it serves as a deviation corrector. Lastly, we validate the effectiveness of our proposed approach by conducting dynamic simulations using the CRANE biped robot. Through comprehensive assessments, including singularity analysis, constraint effectiveness evaluation, and walking experiments on discrete terrains, we demonstrate the superiority and practicality of our approach in enhancing walking performance while ensuring safety. Overall, our research contributes to the advancement of biped robot locomotion by addressing gait optimisation from multiple perspectives, including singularity handling, safety constraints, and deviation learning.

Keywords: Constraints’ handling, deep deterministic policy gradient, deviation learning, reinforcement learning, gait optimisation

DOI: 10.3233/ICA-230724

Journal: Integrated Computer-Aided Engineering, vol. 31, no. 2, pp. 139-156, 2024