RL-赵-(二)-基于模型：贝尔曼/Bellman公式04【贝尔曼方程的矩阵-向量形式（Matrix-Vector Form）】【从Bellman方程求解State Value：①闭式解、②迭代解】2.6 贝尔曼方程的矩阵-向量形式（Matrix-Vector Form of the Bellman Equation）vπ(s)=E[Rt+1∣St=s]+γE[Gt+1∣St=s]=∑a∈Aπ(a∣s)∑r∈Rp(r∣s,a)r⏟meanofimmediaterewards+γ∑a∈Aπ(a∣s)∑s′∈Sp(s′∣s,a)vπ(s′)⏟meanoffuturerewards=∑a∈Aπ(a∣s)[∑r∈Rp(r∣s,a)r+γ∑s′∈Sp(s′∣s,a)vπ(s′)],foralls∈S(2.7)\begin{aligned}\color{red}{v_{\pi}(s)}=\mathbb{E}[R_{t+1}\mid S_t=s]+\gamma\mathbb{E}[G_{t+1}\mid S_t=s]\\=\underbrace{\sum_{a\in\mathcal{A}}\pi(a\mid s)\sum_{r\in\mathcal{R}}p(r\mid s,a)r}_{\text{mean of immediate rewards}}+\underbrace{\gamma\sum_{a\in\mathcal{A}}\pi(a\mid s)\sum_{s'\in\mathcal{S}}p(s'\mid s,a)v_{\pi}(s')}_{\text{mean of future rewards}}\\=\sum_{a\in\mathcal{A}}\pi(a\mid s)\left[\sum_{r\in\mathcal{R}}p(r\mid s,a)r+\gamma\sum_{s'\in\mathcal{S}}p(s'\mid s,a)v_{\pi}(s')\right],\quad\text{for all }s\in\mathcal{S}\end{aligned}\qquad (2.7)vπ​(s)​=E[Rt+1​∣St​=s]+γE[Gt+1​∣St​=s]=meanofimmediaterewardsa∈A∑​π(a∣s)r∈R∑​p(r∣s,a)r​​+meanoffuturerewardsγa∈A∑​π(a∣s)s′∈S∑​p(s′∣s,a)vπ​(s′)​​=a∈A∑​π(a∣s)[r∈R∑​p(r∣s,a)r+γs′∈S∑​p(s′∣s,a)vπ​(s′)],foralls∈S​(2.7)式 (2.7) 中的贝尔曼方程采用逐元素形式（Element-Wise Form）表示。由于它对每个状态都成立，因此可以将所有方程组合起来，用矩阵-向量形式简洁表示。该形式会经常用于分析贝尔曼方程。为了推导矩阵-向量形式，首先将贝尔曼方程：vπ(s)=E[Rt+1∣St=s]+γE[Gt+1∣St=s]=∑a∈Aπ(a∣s)∑r∈Rp(r∣s,a)r⏟meanofimmediaterewards+γ∑a∈Aπ(a∣s)∑s′∈Sp(s′∣s,a)vπ(s′)⏟meanoffuturerewards=∑a∈Aπ(a∣s)[∑r∈Rp(r∣s,a)r+γ∑s′∈Sp(s′∣s,a)vπ(s′)],foralls∈S(2.7)\begin{aligned}\color{red}{v_{\pi}(s)}=\mathbb{E}[R_{t+1}\mid S_t=s]+\gamma\mathbb{E}[G_{t+1}\mid S_t=s]\\=\underbrace{\sum_{a\in\mathcal{A}}\pi(a\mid s)\sum_{r\in\mathcal{R}}p(r\mid s,a)r}_{\text{mean of immediate rewards}}+\underbrace{\gamma\sum_{a\in\mathcal{A}}\pi(a\mid s)\sum_{s'\in\mathcal{S}}p(s'\mid s,a)v_{\pi}(s')}_{\text{mean of future rewards}}\\=\sum_{a\in\mathcal{A}}\pi(a\mid s)\left[\sum_{r\in\mathcal{R}}p(r\mid s,a)r+\gamma\sum_{s'\in\mathcal{S}}p(s'\mid s,a)v_{\pi}(s')\right],\quad\text{for all }s\in\mathcal{S}\end{aligned}\qquad (2.7)vπ​(s)​=E[Rt+1​∣St​=s]+γE[Gt+1​∣St​=s]=meanofimmediaterewardsa∈A∑​π(a∣s)r∈R∑​p(r∣s,a)r​​+meanoffuturerewards