三、Summary of the storySummary of the story首先从一个objective function出发J ( w ) = E [ ( v π ( S ) − v ^ ( S , w ) ) 2 ] \begin{aligned}J(w)=\mathbb{E}[(v_\pi(S)-\hat{v}(S,w))^2]\end{aligned}J(w)=E[(vπ​(S)−v^(S,w))2]​这个目标函数表明这是一个policy evaluation问题.然后对这个objective function进行优化，优化方法使用gradient-descent algorithm:w t + 1 = w t + α t ( v π ( s t ) − v ^ ( s t , w t ) ) ∇ w v ^ ( s t , w t ) w_{t+1}=w_t\:+\alpha_t(v_\pi(s_t)-\hat{v}(s_t,w_t))\nabla_w\hat{v}(s_t,w_t)wt+1​=wt​+αt​(vπ​(st​)−v^(st​,wt​))∇w​v^(st​,wt​)但是问题是里边有一个v π ( s t ) v_{\pi}\left(s_{t}\right)v