1

I'm reading a research paper on generating/synthesizing videos:
MoCoGAN: Decomposing Motion and Content for Video Generation

To evaluate the generated videos, they have used a metric called 'Average Content Distance'. I couldn't find any material on google related to this. Can anyone please explain what Average Content Distance means?

Here is the snippet from the paper

we first computed the average color of the generated shape in each frame. Each frame was then represented by a 3-dimensional vector. The ACD is then given by the average pairwise L2 distance of the per-frame average color vectors.

What I understood from this is as follows:
For each frame, convert rgb to gray (average of color). Then for successive frame, calculate the l2 distance. $$\frac{1}{MN} \sum_{x=1}^{M}\sum_{y=1}^{N}{(Frame_i(x,y) - Frame_{i+1}(x,y))^2}$$ This gives ACD. Have I understood it correctly?

Also, how does this metric represents quality of a video? How can this be used to compare qualities of different generated videos? You can also point me towards some references.

Thanks!

Nagabhushan S N
  • 724
  • 3
  • 8
  • 23

1 Answers1

3

This article presents 2 ways to implement ACD metric. You talk about ACD-I (using the article's notation). As I understood,

  • you first average colors, i.e. sum over pixels in the image plane, for each color in all frames: $\mathrm{avg}_i = \frac{1}{MN}\sum_{x,y}\mathrm{Frame}_i $.
  • then, in the resulting 3D vectors for every two consecutive ones, you calculate L2 distance: $d_i = \sqrt{\sum_{l=1}^3(\mathrm{avg}_{i,l} - \mathrm{avg}_{i+1,l})^2}$. In general, you might want to use other distances, which is not prohibited.
  • sum it up (since we take into account all frames we have, not just one pair) and divide by the number of frames (because the metric should not depend on it [at least, I suppose so], otherwise longer videos will have larger metric values): $\mathrm{ACD} = \frac{1}{K-1}\sum_i d_i$ (if there are K frames).

ACD-C is obtained in the same way but instead frames you use feature vectors extracted from images (frames) with 'encoding-like' network. OpenFace probably is a good choice when dealing with facial expressions.

Your formula might work differently from the authors' intention. Imagine a white spot on a black screen. From frame to frame that spot is gradually moving from one side to the other. Your metric shows that the content in this video is changing. And the faster the spot moves, the greater changes happen (ok, in case of a spot and a black screen at a certain speed the saturation point will be reached and your metric will stop changing [when in one frame the spot moves a distance equal to its diameter], but that's another story). However, in fact, there are no changes in the content. You still have the spot and the black screen. This is why you need averaging (not summing 'changes' and making it independent from the image size - this is what your formula does)

Here you can see the implementation of ACD metric(s). I cannot vouch that is 100% correct. So, let me know if there are any kind of uncertainties.

Vlad
  • 46
  • 3
  • Thank you! Your explanation made it clear. Now I understand it properly. We extract a feature vector from each frame which represents the content and then take the l2 distance between the successive vectors.

    ACD was originally introduced in MoCoGAN paper. The paper you have referred to has extended it.

    And the code you referred to, I couldn't understand it 100%, but I got the idea. Thanks a lot

     – Nagabhushan S N May 27 '19 at 13:10
  • https://stackoverflow.com/a/56377648/3337089 – Nagabhushan S N May 30 '19 at 11:42