Reinforcement Learning for Adversarial Robot Soccer

Training Pipeline

The training pipeline for individual soccer player is straight-forward. I built a ROS2 physics simulator that simulates the interaction between the robot and the soccer, and the robot against its opponent or teammates. The DDPG RL algorithm is updated by sampling collected experience tuple composed of actions, rewards, and previous plus current states.

Training Attacker

I trained the naive attacker first and then proceed to train a defender, and then continue this cycle multiple times to make both the attacker and defender more sophisticated. The naive attacker's goal is simple--dribble the ball through the opening at the other end of the field. I gave the robot reward for each step towards the goal without taking it outside the field, and punish the agent when the ball is out of field or taken to a wrong direction. The robot is given different starting positions at each reset so it will know how to attack no matter its current positions.

Training the Defender

After training the attacker, I proceed to train a defender that can tackle the ball from the attacker. The attacker is controlled by the trained model, while the defender's control network is being trained from scratch.

ROS2 Simulator

To visualize the training process and evaluate the trained model, I created this physics simulator based on ROS2 that features basic ball physics (velocity decay included). The turtlebot robot models receive speed, direction, kicking power, and kicking direction.