7

There are two agents $i=1,2$. The state $k$ is governed by $\tau_i\in[0,1]$ where \begin{align} \dot{k} = f(k,\tau_1,\tau_2). \end{align}

Define the value function of player $i$ by \begin{align} v_i(k) := \sup_{\tau_i}\int^\infty_0{e^{-\rho t}F(k,\tau_i,\sigma_{-i})dt}. \end{align} where $\sigma_{-i}(k)$ is the policy function or a markovian stratgey profile of player $-i$. And $\rho>0$ is the time preference rate.

The Hamilton-Jacobi-Bellman equation reads \begin{align} \rho v_i(k) = \sup_{\tau_i}\{F(k,\tau_i,\sigma_{-i}) + v'(k)f(k,\tau_i,\sigma_{-i})\}. \end{align}

Now assume I approximated a function $\hat{v}_i$ by value function iteration. How do I know that it's the true value function?

clueless
  • 1,579
  • 10
  • 21
  • Check the distance with the last iteration. If you like it, you can stop. If it's too big, keep iterating. – VicAche Jun 05 '15 at 09:17
  • first of all you can solve an analytical example and then test your algorithm against it. Second, the value function is a fixed point, so you can google for some tests of convergence to a fixed point, at least one heurisitic rules comes to mind, the distance between i and i+1 is decreasing , strict concavity ensures global optimality – user157623 Jun 08 '15 at 12:20

1 Answers1

1

The first panel shows the final value function after iterating 80k (!). Should probably try an implicit method. The second panel shows the evolution from frist guess to final result $v^0\to v^j$, $j=80k$. Third panel is the distance between the value functions and fourth shows the error.

Note that the value function is convex-concave. This is due to the fact that for the limit case the law of motion tends to infinty $\lim_{k\to 0} \dot{k} = \infty$.

By the maximum principle optimality requires that the Hamiltonian \begin{align} H_1(k,\tau_1,\tau_2^*,\lambda_1) := F(k,\tau_1,\tau_2^*) + \lambda_1 f(k,\tau_1,\tau_2^*) \end{align} is concave in $(k,\tau_1,\tau_2)$ (sufficient condition). Now we have
\begin{align} \rho v(k) = \max_{\tau_1} H_1(k,\tau_1,\tau_2^*,\lambda_1). \end{align}

  • I was wondering if the value function must be (strictly) concave to ensure optimality?

Any reference is appreciated.

vfi

clueless
  • 1,579
  • 10
  • 21