Asynchronous Deep Reinforcement Learning

Asynchronous Deep Reinforcement Learning

Comments 0 Comment

Deep reinforcement learning saw an explosion in the mid 2010s due to the development of the deep q learning (DQN) algorithm. DQN’s success was driven the use of multiple innovations. Perhaps the most important being the use of experience replay for updating deep neural networks . While this was a breakthrough in the field, it isn’t without its drawbacks.. For one, it requires a non trivial amount of ram to store the million or so experiences from the agent….

Read More Read More

Make An Experience Replay Buffer In One Line Of Code

Make An Experience Replay Buffer In One Line Of Code

Comments 0 Comment

This article is for those of us who have gotten stuck implementing an experience replay buffer as a beginner. That includes me. I see a lot of DRL agent implementations online using experience replay buffers made out of a bunch of numpy arrays. This data strategy isn’t new, and there’s a reason it runs nice and fast (https://en.wikipedia.org/wiki/AoS_and_SoA). It is actually a pretty good solution if your batch sizes are big. So you should definitely just copy and paste it…

Read More Read More

How to learn hard things
The Art of Learning

The Art of Learning

Comments 0 Comment

When we think of what it means to learn, we often reference our experiences in school. We equate learning to being spoon fed information from an authority figure, diligently copying down the salient nuggets for future reference. Those nuggets are harvested weeks later, in preparation for an “exam” that is meant to test our learning, but typically only tests our ability to regurgitate what we were “taught”. Shortly after the exam, the information is forgotten, necessitating urgent cramming sessions before…

Read More Read More

A Conceptual Introduction to Policy Gradient Methods

A Conceptual Introduction to Policy Gradient Methods

Comments 0 Comment

Reinforcement learning algorithms tend to fall into two distinct categories: value based and policy based learning. Q Learning, and its deep neural network implementation, Deep Q Learning, are examples of the former. Policy gradient methods, as one might guess from the name, are examples of the latter. While value based methods have shown performance that meets or exceeds human level play in a variety of environments, they suffer from some significant drawbacks that limit their viability across the broader array…

Read More Read More

Fundamental Concepts in Reinforcement Learning

Fundamental Concepts in Reinforcement Learning

Comments 1 comment

We’ve touched on reinforcement learning many times here, as it represents our best chance at developing something approximating artificial general intelligence. We’ve covered everything from Monte Carlo methods, to Deep Q Learning, to Policy Gradient methods, using both the Pytorch and Tensorflow frameworks. What we haven’t discussed on this channel is the what and the how of reinforcement learning. That oversight ends today. Let’s get started. Essential concepts You’re probably familiar with supervised learning, which has been successfully applied to…

Read More Read More

Designing Your Own Open AI Gym Compatible Reinforcement Learning Environment

Designing Your Own Open AI Gym Compatible Reinforcement Learning Environment

Comments 7 comments

The Open AI gym provides a wide variety of environments for testing reinforcement learning agents, however there will come a time when you need to design your own environment. Perhaps you are designing an inventory management system, or even creating an agent to perform real time bidding in search auctions. Whatever the use case, you will have to design your own environment, as there aren’t any off the shelf solutions that lend themselves to these tasks. It may seem a…

Read More Read More

Coding a Deep Q Network in PyTorch

Coding a Deep Q Network in PyTorch

Comments 9 comments

In a previous post we covered a quick and dirty introduction to deep Q learning. This covered the conceptual basics: an agent uses a deep neural network to approximate the value of its action-value function, and attempts to maximize its score over time using an off-policy learning strategy. The high level intuition is sufficient to know what’s going on, but now it’s time to dive into the details to actually code up the deep Q network. In this post, we’re…

Read More Read More

A Quick Introduction to Deep Q Learning

A Quick Introduction to Deep Q Learning

Comments 4 comments

Several years ago the Deep Mind team announced that they had designed a new reinforcement learning algorithm capable of beating human level play in multiple games from the Atari library. This new algorithm, deep Q learning, was the marriage of deep neural networks and traditional Q learning. Rather than being a trivial bolt on, deep Q learning is actually an ingenious solution to a difficult problem: how to handle the incredibly large state spaces of even primitive video games? For…

Read More Read More