agent naming environment

In this project we create an agent able to distinguish and name object on the environment

Breakout

Similar to keras article we create a model to parse the input image from atari games and decide how to interact with the environment: i.e. how to move.

using openai gym to train an agent to play breakout

To understand how the agent is learning the enviroment we create a visualization to focus on the main step of learning.

visualization to monitor the development of the learning process

The visualization consists in:

• scores: loss and reward
• inputs: batch average of the states (frames) and velocity (frame difference)
• actions: distribution of moves over time
• attention map: where the model is focusing the attention

Based on the reward (bricks crushed by the ball) the agent learns how to behave in such environment

implementation

Compared to the original implementation of the game we performed few differences:

• velocity and collision (velocity change) instead of a stack of 4 consecutive images (dropped acceleration because rearly happens on the same pixel)
• batch of 64 consecutive states (I didn’t get the advantage of randomizing) and use of deque
• treaning each 16 actions
• genetic selection of algorithm considering the fitness
• feedback loop between actions (actions are both input and output of the model with -1 time difference)
• output classes based on break down of the interaction function

For more references

velocity and collision as input parameters

The model outputs different classes defined by measurable interaction with the environment, mainly: myself, tool, walls, info, dangers…

attention

The attention map tells a lot about the key challanges of this exercise

• who am I?: different parts are moving, what is the part of the picture I control?
• tools: what parts of the environemnt I can use
• walls: what parts define the boundary of the environment
• danger: what parts harm my entity
• reward: what is a benefit for me
• info: what is the outside of the environment just providing information

Compared to other games breakout has a lot of interesting challanges in the understanding of causalit y.

attivation classes

We do the same for the different types of moves

causality

In this game you get the reward the moment the ball hits the bricks but you control only the paddle. In the eyes of the agent I first see a clear correlation:

• score: for each reward a digit change: 100% correlation, 0% causality
• hit: for each reward a brick disappears: 100% correlation, long rage causality -> effect
• ball: multiple bounces -> mean
• paddle: complete control through actions -> cause

evolution of attention over time, the focus on digits drops over time, the agent is then learning to avoid losing life focusing on the bottom.

We insert additional scores as output which will depend directly from an action and study the activation for that specific action: