Deep reinforcement learning hands-on : apply modern RL methods, with deep Q-networks, value iteration, policy gradients, TRPO, AlphaGo Zero and more / Maxim Lapan.

Format:

Book

Author/Creator:

Lapan, Maxim, author.

Language:

English

Subjects (All):

Reinforcement learning.

Physical Description:

1 online resource (1 volume) : illustrations

Edition:

1st edition

Place of Publication:

Birmingham, England : Packt Publishing, 2018.

System Details:

text file

Summary:

This practical guide will teach you how deep learning (DL) can be used to solve complex real-world problems. About This Book Explore deep reinforcement learning (RL), from the first principles to the latest algorithms Evaluate high-profile RL methods, including value iteration, deep Q-networks, policy gradients, TRPO, PPO, DDPG, D4PG, evolution strategies and genetic algorithms Keep up with the very latest industry developments, including AI-driven chatbots Who This Book Is For Some fluency in Python is assumed. Basic deep learning (DL) approaches should be familiar to readers and some practical experience in DL will be helpful. This book is an introduction to deep reinforcement learning (RL) and requires no background in RL. What You Will Learn Understand the DL context of RL and implement complex DL models Learn the foundation of RL: Markov decision processes Evaluate RL methods including Cross-entropy, DQN, Actor-Critic, TRPO, PPO, DDPG, D4PG and others Discover how to deal with discrete and continuous action spaces in various environments Defeat Atari arcade games using the value iteration method Create your own OpenAI Gym environment to train a stock trading agent Teach your agent to play Connect4 using AlphaGo Zero Explore the very latest deep RL research on topics including AI-driven chatbots In Detail Recent developments in reinforcement learning (RL), combined with deep learning (DL), have seen unprecedented progress made towards training agents to solve complex problems in a human-like way. Google’s use of algorithms to play and defeat the well-known Atari arcade games has propelled the field to prominence, and researchers are generating new ideas at a rapid pace. Deep Reinforcement Learning Hands-On is a comprehensive guide to the very latest DL tools and their limitations. You will evaluate methods including Cross-entropy and policy gradients, before applying them to real-world environments. Take on both the Atari set of virtual games and family favorites such as Connect4. The book provides an introduction to the basics of RL, giving you the know-how to code intelligent learning agents to take on a formidable array of practical tasks. Discover how to implement Q-learning on ‘grid world’ environments, teach your agent to buy and trade stocks, and find out how natural language models are driving the boom in chatbots. Style and approach Deep Reinforcement Learning Hands-On explains the art of building self-learning agents using algori...

Contents:

Cover

Copyright

Packt upsell

Contributors

Table of Contents

Preface

Chapter 1 - What is Reinforcement Learning?

Learning - supervised, unsupervised, and reinforcement

RL formalisms and relations

Reward

The agent

The environment

Actions

Observations

Markov decision processes

Markov process

Markov reward process

Markov decision process

Summary

Chapter 2 - OpenAI Gym

The anatomy of the agent

Hardware and software requirements

OpenAI Gym API

Action space

Observation space

Creation of the environment

The CartPole session

The random CartPole agent

The extra Gym functionality - wrappers and monitors

Wrappers

Monitor

Chapter 3 - Deep Learning with PyTorch

Tensors

Creation of tensors

Scalar tensors

Tensor operations

GPU tensors

Gradients

Tensors and gradients

NN building blocks

Custom layers

Final glue - loss functions and optimizers

Loss functions

Optimizers

Monitoring with TensorBoard

TensorBoard 101

Plotting stuff

Example - GAN on Atari images

Chapter 4 - The Cross-Entropy Method

Taxonomy of RL methods

Practical cross-entropy

Cross-entropy on CartPole

Cross-entropy on FrozenLake

Theoretical background of the cross-entropy method

Chapter 5 - Tabular Learning and the Bellman Equation

Value, state, and optimality

The Bellman equation of optimality

Value of action

The value iteration method

Value iteration in practice

Q-learning for FrozenLake

Chapter 6 - Deep Q-Networks

Real-life value iteration

Tabular Q-learning

Deep Q-learning

Interaction with the environment

SGD optimisation

Correlation between steps

The Markov property

The final form of DQN training

DQN on Pong

Wrappers.

DQN model

Training

Running and performance

Your model in action

Chapter 7 - DQN Extensions

The PyTorch Agent Net library

Agent

Agent's experience

Experience buffer

Gym env wrappers

Basic DQN

N-step DQN

Implementation

Double DQN

Results

Noisy networks

Prioritized replay buffer

Dueling DQN

Categorical DQN

Combining everything

References

Chapter 8 - Stocks Trading Using RL

Trading

Data

Problem statements and key decisions

The trading environment

Models

Training code

The feed-forward model

The convolution model

Things to try

Chapter 9 - Policy Gradients - An Alternative

Values and policy

Why policy?

Policy representation

Policy gradients

The REINFORCE method

The CartPole example

Policy-based versus value-based methods

REINFORCE issues

Full episodes are required

High gradients variance

Exploration

Correlation between samples

PG on CartPole

PG on Pong

Chapter 10 - The Actor-Critic Method

Variance reduction

CartPole variance

Actor-critic

A2C on Pong

A2C on Pong results

Tuning hyperparameters

Learning rate

Entropy beta

Count of environments

Batch size

Chapter 11 - Asynchronous Advantage Afctor-Critic

Correlation and sample efficiency

Adding an extra A to A2C

Multiprocessing in Python

A3C - data parallelism

A3C - gradients parallelism

Chapter 12 - Chatbots Training with RL

Chatbots overview

Deep NLP basics

Recurrent Neural Networks

Embeddings

Encoder-Decoder.

Training of seq2seq

Log-likelihood training

Bilingual evaluation understudy (BLEU) score

RL in seq2seq

Self-critical sequence training

The chatbot example

The example structure

Modules: cornell.py and data.py

BLEU score and utils.py

Model

Training: cross-entropy

Running the training

Checking the data

Playing with the trained model

Training: SCST

Running the SCST training

Telegram bot

Chapter 13 - Web Navigation

Web navigation

Browser automation and RL

Mini World of Bits benchmark

OpenAI Universe

Installation

Actions and observations

Environment creation

MiniWoB stability

Simple clicking approach

Grid actions

Example overview

Starting containers

Training process

Checking the learned policy

Issues with simple clicking

Human demonstrations

Recording the demonstrations

Recording format

Training using demonstrations

TicTacToe problem

Adding text description

Chapter 14 - Continuous Action Space

Why a continuous space?

Environments

The Actor-Critic (A2C) method

Using models and recording videos

Deterministic policy gradients

Recording videos

Distributional policy gradients

Architecture

Chapter 15 - Trust Regions - TRPO, PPO and ACKTR

Introduction

Roboschool

A2C baseline

Videos recording

Proximal Policy Optimisation

Trust Region Policy Optimisation

A2C using ACKTR

Chapter 16 - Black-Box Optimization in RL

Black-box methods.

Evolution strategies

ES on CartPole

ES on HalfCheetah

Genetic algorithms

GA on CartPole

GA tweaks

Deep GA

Novelty search

GA on Cheetah

Chapter 17 - Beyond Model- Free - Imagination

Model-based versus model-free

Model imperfections

Imagination-augmented agent

The environment model

The rollout policy

The rollout encoder

Paper results

I2A on Atari Breakout

The baseline A2C agent

EM training

The imagination agent

The I2A model

The Rollout encoder

Training of I2A

Experiment results

The baseline agent

Training EM weights

Training with the I2A model

Chapter 18 - AlphaGo Zero

Board games

The AlphaGo Zero method

Overview

Monte-Carlo Tree Search

Self-play

Training and evaluation

Connect4 bot

Game model

Implementing MCTS

Testing and comparison

Connect4 results

Book summary

Other Books You May Enjoy

Index.

Notes:

"Expert insight."

Includes bibliographical references and index.

Description based on print version record.

ISBN:

9781788839303

1788839307

OCLC:

1046682461