Simulating a Hamilton-Jacobi-Bellman

Question

Say I have solved an HJB of the form:

$\rho V(k) = \max_c g(c) + V'(k)(z - c)$

I have calibrated $\rho$ to monthly parameters. I would like to simulate the development of $k$. I start with $k(0)$. However, unlike in discrete-time, I'm not sure what happens next.

Is $k(1) = k(0) + (z-c(0))$, with $k(1)$ being the value of $k$ for the next month? Or do I need to somehow integrate along the lines of $k(1) = \int_0^1 (z-c(t))dt + k(0)$? There's many references on solving these things, but I couldn't find any references on simulating it.

Answer 1

Expanding on jmbejara's answer and your question about integration, there are really three steps here:

1. Solve the HJB equation $\rho V(k) = \max_c g(c)+V'(k)(z-c)$. (You're already done this.)
2. Obtain the policy function $c(k)=\text{argmax}_c g(c)+V'(k)(z-c)$. (You presumably already did this while solving the HJB.)
3. Substitute $c(k)$ into the law of motion for $k$ and numerically solve that differential equation for $k(t)$:$$\dot{k}=z-c(k)$$

It sounds like you already had (1) and (2) and were just unsure about what exactly to do in (3). If you're working in a deterministic environment (and from the HJB it looks like you are), no discretization on your part is necessary: you can just use a canned ODE solver from Matlab or any other software package to numerically solve the differential equation above for $k(t)$.

To be a little more specific:

• If the HJB is simple and you have an analytical solution for $V(k)$, and thus can easily compute $c(k)$ as well, that's great. You can just plug $c(k)$ into the law of motion for $k$ and tell your software package of choice to solve it and obtain the trajectory of $k(t)$.
• If you had to numerically solve the HJB, then that's great because it means that you probably have already used some kind of numerical ODE solution technique and will be familiar with it. On the minus side, it means that when you initially obtain $V(k)$ and $c(k)$, you'll only get it for a grid of $k$, rather than having some kind of formula you can apply everywhere - which is a problem, because you'll need to be able to determine $c(k)$ at any point $k$ in a continuum if you're going to numerically solve the $\dot{k}$ differential equation for $k(t)$. The simplest way around this, to my knowledge, is to fit a spline to (or use some other approximation method on) the discrete grid of $(k_{grid},c(k_{grid}))$ values that you have. Then you can obtain an approximate value of $c(k)$ anywhere that's covered by the spline, and it will be extremely accurate assuming that your ODE is well-behaved and you have a dense grid.

Things get more complicated in a stochastic environment, but your case looks deterministic. (And to be clear, I'm just a dabbler here: numerical methods are not my area of expertise. But this is what I know.)

Answer 2

You work in discrete time, see for example this resource.

Or this (incomplete) paper, where the same author participates. In the beginning the paper talks about "coarse discrete methods" and may give the impression that it contains some continuous time simulation method. It really doesn't -see the description of their algorithm in the Appendix, page 30.

Also this MPhil dissertation, which, being a graduate thesis, describes the steps in some detail, and with a wealth of referenced literature.

Answer 3

Since Ben's notes were already mentioned I'm not quite sure why you don't iterate over time using a forward approximation as he suggests \begin{align} \dot k(t) :\approx \frac{k(t+1) - k(t)}{\Delta t} \Leftrightarrow k(t+1) \approx \Delta t \dot k(t) + k(t) \end{align}

which gives \begin{align} k(1) \approx \Delta t (z-c(0)) + k(0) \end{align}

Answer 4

Once you have solved the equation $\rho V(k) = \max_c g(c) + V'(k)(z - c)$ to find the function $V(k)$ you should be able to solve for what is called the policy function. The function $V(k)$ is the value function. Knowing $V(k)$ should then give you the information that you need to solve for the policy, say $c(k,t)$. This will then tell you how $k$ will evolve over time.