Performing Image alignment using pyramid levels with semi dense depth

Question

To gain some confidence, I want to implement the camera tracking (optimization problem) discussed in Semi Dense Visual Odometry for a monocular camera^{J Engel, J Strum, D Cremers}

$$E(\xi) = \underset{i}\Sigma\frac{\alpha(r_i(\xi))}{\sigma_{d_i}^2}(r_i(\xi))$$ $$r_i(\xi) = (I_2(w(x_i, d_i, \xi)) - I_1(x_i))$$

Using Gauss-Newton method, as discussed in the same paper, (local optima) $\xi$ between two monocular images can be found. Here is how I think the process goes:

Given

• Start with initial guess $\xi_0$ (~ 0), Images $I_1, I_2$, Depth for some points in image $I_1$ is given

Steps

• Start at lowest level (most coarse) $L_n$ pyramid image (same as shrinked image?) and corresponding depth map (for points in shrinked image?)

• Gauss Newton iterations with some library (I plan to use python for ease).

  • Setup the residual calculation function that takes input $I_2$, $I_1$, {$x_i$}, {$d_i$} and {$\xi$} and produces $r_i$. The Jacobian involved will be calculated numerically.
  • I can skip $\alpha(.)$ and $\sigma_{d_i}^2$ for now. $E(\xi) = \underset{i}\Sigma ~r_i^2$
  • At the end I expect to get the result $\xi_L$

• Use the solution $\xi_L$ as initial guess and repeat for the next pyramid level (n-1)

Questions:

• Am I missing some step in above process ? Please let me know.

• Is there a library function in openCV that take an image (and its depth image) as input and give in output the requested pyramid level (for a choosen n) image as output (depth image will also need to be shrinked)?

PS: can someone with higher reputation add the tag "Image-alignment" to this question?

Answer 1

I'm going to try to answer this question but please don't flame me if I got something wrong. Those were two heavy papers and I didn't have as much time as I wish to go through them.

---

The pyramids are the same as shrank image indeed. Image pyramids are images with lower resolution. As a general concept, by reducing the resolution and representing a different amount of details, the tracking algorithm can focus on different types of feature and become more generic. See pyrUp/Down in OpenCV.

---

It is a little hard to answer because it is not clear what is the granularity of your question.

I feel like the general steps of the algorithm are there but if I have the time later, I'll go through it again later. However, Gaussian-Newton optimization are rarely plug and play for me, so I would be cautious there in there of the time needed to have it running.

I'm wondering if, since you'll be ignoring $\alpha$ and $\sigma^2_{d_{i}}$ You're not implementing the method of Steinbrücker et al. in Real-Time Visual Odometry from Dense RGB-D Images (full disclaimer I didn't have time to read this one in details just skimmed it).