Creating your first DeepRacer model is easy, but what is DeepRacer?

AWS DeepRacer allows developers of any skill level to get started with machine learning with hands-on tutorials and guidance on building reinforcement learning models. Reinforcement learning is a branch of machine learning, ideal for a variety of practical business problems from robotics automation, to finance, to game optimization, and autonomous vehicles. - AWS DeepRacer [1]

Creating an AWS account

The first step is to create an account on AWS. You'll need to provide some basic information to get started.

Note: It is highly recommended that you set up two-factor authentication on your root account. Follow the AWS guide to get multi-factor setup. It is also highly recommended to set up a free tier AWS budget alert to let you know if you are being charged without realising it.

Head to DeepRacer

Once you have your account set up, search for DeepRacer in the services list. On the left click on the Your Models and then click the orange Create Model at the top right. This will present you with the model creation screen below.

Give it an appropriate name and a description if you like and then select a track to train on. For your first model I would highly suggest keeping it simple and using the re:Invent 2018 track shown below.

Once selected progress to the next screen where you'll pick the race type and model type. For now, leave it set to time trial, leave the model type set to PPO and leave the parameters alone. You'll have plenty of time to tweak these in future but I suggest getting started with the basics first.

Creating the action space

On the next screen, you'll be asked to create the action space. Think of the action space as all the possible things the car can decide to do. It might choose to turn left sharp and go fast, but it might also choose to go slow in a straight line.

Discrete vs Continuous

By default, the model is set to continuous which means the model can choose to travel anywhere between the minimum and maximum speed and turn any angle between the minimum and maximum steering angle. With discrete, you get to pick specific combinations of speeds and turning angles you want to allow.

I suggest leaving it on continuous but upping the maximum speed to 3 and changing the minimum and maximum steering angles to -20 and 20 respectively.

Creating your first reward function

You'll now be faced with some Python code which is a default implementation of a reward function as shown below

So, what is a reward function?

Imagine you're teaching your dog new tricks, you can't say sit because it doesn't understand you. Instead, you tell it to sit and when it does, you reward it. Eventually, through enough training, your dog will sit most of the time when prompted because it associates its actions with getting treats. It's never perfect, and some people are better at training dogs than others. Your reward function acts as the treat for your car.

You have access to multiple parameters such as speed, location, track width, distance from centre as well as several others and it's your job to create a model which you think appropriately rewards the car for doing the right thing.

Feel free to leave the model as AWS created it, or make some changes. Don't overcomplicate things to start with. My first model took the speed and whether the car was off the track.

def reward_function(params):
  speed = params["speed"]
  on_track = params["all_wheels_on_track"]

  # Reward the car more at higher speeds
  reward = speed * speed

  # Penalise the car if it goes off track
  if not on_track:
    reward = reward * 0.01

  return float(reward)

Check out the full list of parameters available to you. You can create all sorts of models which will be able to tackle all sorts of track layouts. Start simple though and work your way up to more complicated designs.

Training the model

Now you've written your reward function, set a training time. I'd suggest training the model for an hour or two initially. At the time of writing AWS offers every new DeepRacer account 10 free training hours, however, this may change so please check beforehand.

You can also choose to enter the race if you like, up to you if you feel brave enough to enter on your first attempt. I think you have limited entries per race though so you may want to wait and create a better model before submitting.

With your model created, AWS will spin up an environment in which to train and evaluate your model. Once it's running you'll see a screen like below.

The graph on the left tells you how well your model is doing. The green line (with circles) shows the average amount of reward gained by the car on that set of runs. The blue line (with diamonds) shows how much of the track the car manages to complete without leaving the track during the training phase. The red line (with squares) then shows the evaluation phase and how much of the track it completes.

Ideally, your graph should have all lines increasing, although you will definitely have times when the model gets worse. This is the fun of creating the model, tweaking the action space and changing the hyperparameters.

Note: in time you'll want to watch out for overtrained models and you can choose to stop training at any point by clicking the Stop Training button at the top right of the page.

Evaluating the model

Once the model has finished training you can then run an evaluation on it, picking 3 or 5 lap evaluations. This will show you the number of times the car went off track, penalties applied, lap times etc.

(Ignore my lap time, try and aim for a 10 - 12 second lap time for the re:Invent 2018 track, the lap record stands at around 8 seconds I think).

Once you're happy with your model you can either have it loaded into a real AWS DeepRacer car or enter the model into a virtual race which can be done through the UI. Winners of these events can then be invited to enter their models in real-world races.

Next step: Enjoy

If you're like me, you'll be addicted in no time. It's a lot of fun coming up with new reward functions and watching your car go around the track. I look forward to seeing you all smashing the leaderboards.

References

[1] Amazon Web Services. (n.d.). DeepRacer Console. Retrieved July 9, 2023, from us-east-1.console.aws.amazon.com/deepracer/..