Reinforcement Learning (Learning)

1 Concepts

1.1  State

The status of the agent with respect to the environment. Grid example, the location of the agent is state.

State space. The set of all states. S

1.2 Action

For each state, there are several possible actions. Grip dxample : 4 movements.

1.3 State transition

When taking an action, the agent moves from one state to another state. Such a process is called state transition.

$$ S_1 \xrightarrow{a_2} S_2 $$

State transition probability: 

$$ p(s_2 | s_1, a_2) = 1 $$

$$ p(s_i | s_1, a_2) = 0 \quad \forall i \neq 2 $$

1.4 Policy

Policy tells the agent what actions to take at a state. 

$$ \pi(a_1 | s_1) = 0 $$

1.5 Reward

Reward is a real number. Positive is encourgement, negative is punishment. 

Mathematical description : conditional probability 

1.6 Trajectory

A trajectory is a state-action-reward chain. 

$$s_1 \xrightarrow[a_2]{r=0} s_2 \xrightarrow[a_3]{r=0} s_5 \xrightarrow[a_3]{r=0} s_8 \xrightarrow[a_2]{r=1} s_9$$

1.7 Return

It is the result of trajectory. 

1.8 Discounted return

\begin{align*}
\text{discounted return} &= 0 + \gamma 0 + \gamma^2 0 + \gamma^3 1 + \gamma^4 1 + \gamma^5 1 + \dots \\
&= \gamma^3 (1 + \gamma + \gamma^2 + \dots) = \gamma^3 \frac{1}{1 - \gamma}.
\end{align*}

• Gamma is small, it focuses on the closer rewards,
• Gamma is big, it focuses on the future's rewards.

1.9 Episode

When interacting with the environment following a policy, the agent may stop at some terminal states. The resulting trajectory is called an episode or a tiral. The opoosite one is continuing task.

 

2 Bellmen Equation

 

 

 

https://www.bilibili.com/video/BV1sd4y167NS/?spm_id_from=333.788.videopod.episodes&vd_source=8c42b55896f8a24fb4f65b7a3383faa1&p=4