How to Apply the SSIM Measure on RGB Images?

Question

I want to compare the similarity of two images by looking at its structure and colors. I need a measure that takes both structural and color fidelity into account. When I checked the formulation of original paper, it is said only x and y mean and variance is measured. Does SSIM account for structure only or does it take color similarity into account too?

Answer 1

What usually can be done is calculate it per channel and have the average (Or weighted average).

So if you have $ {R}_{1}, {G}_{1}, {B}_{1} $ and $ {R}_{2}, {G}_{2}, {B}_{2} $ you would do:

$$ {SSIM}_{r} \left( {R}_{1}, {R}_{2} \right), {SSIM}_{g} \left( {G}_{1}, {G}_{2} \right), {SSIM}_{b} \left( {B}_{1}, {B}_{2} \right) $$

Then $ {SSIM}_{rgb} = {w}_{r} {SSIM}_{r} + {w}_{g} {SSIM}_{g} + {w}_{b} {SSIM}_{b} $.

For instance, MATLAB's ssim() does the above with $ {w}_{r} = {w}_{g} = {w}_{b} = \frac{1}{3} $.

SciKit Image's compare_ssim() also apply it channel wise using the multichannel flag:

Answer 2

I don't see a commonly used standard when implementing it; the original paper [1] and reference code [2] only provide the definition for grayscale images.

This means the SSIM result for color images may vary from implementation to implementation. The $\frac{1}{3}SSIM_R + \frac{1}{3}SSIM_G + \frac{1}{3} SSIM_B$ answer @Royi gives is how ImageQualityIndexes.jl (Julia version) is implemented.

As for MATLAB's built-in ssim() function, it treats the RGB image as 3D volumic data instead of three independent 2D slices. This is quite different because SSIM internally applies a few filters. If it is 3D volumic data, the filter kernel is 3D, and if it is 2D slices, then the filter kernel is still 2D.

edit ssim in MATLAB and you'll notice this note

A and REF can be arrays of up to three dimensions. All 3D arrays are considered 3D volumetric images. RGB images will also be processed as 3D volumetric images.

Unconditionally treating N-dimensional RGB images as N+1 dimensional gray images is not a good idea because

1. filtering a length-3 dimension almost certainly introduces artifacts due to the padding strategy. For instance, SSIM by default applies a gaussian kernel of size 11, and thus the image is padded to at least size (h+10, w+10, 3+10) before applying the filter.
2. the channel contains quite different information from height/width, and if you try to mix them together, you usually don't get a good result.

This is why ImageQualityIndexes.jl deliberately diverges from MATLAB's numeric results.

---

To simply verify that MATLAB uses 3D volumic definition and ImageQualityIndexes use the mean of channelwise SSIM definition, I use the "peppers_color" and "mandrill" color image from TestImages.jl.

% MATLAB
img1 = imread("peppers_color.tif");
img2 = imread("mandrill.tiff");
% weighted sum of the channelwise ssim
ssim_r = ssim(img1(:,:,1), img2(:,:,1)) % 0.1468
ssim_g = ssim(img1(:,:,2), img2(:,:,2)) % 0.1037
ssim_b = ssim(img1(:,:,3), img2(:,:,3)) % 0.0911
wssim = 1/3 * (ssim_r + ssim_g + ssim_b) % 0.1139
% MATLAB's built-in ssim
ssim(img1, img2) % 0.1219

# julia
using ImageQualityIndexes, TestImages
img1 = testimage("peppers_color")
img2 = testimage("mandrill")
assess_ssim(img1, img2) # 0.11387

References:

• [1] Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4), 600-612.
• [2] Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2003). The SSIM Index for Image Quality Assessment. Retrived May 30, 2019, from http://www.cns.nyu.edu/~lcv/ssim/

Answer 3

It's not indicated for colors, it only accounts for Structure. You could check more info on this paper Understanding SSIM.

An alternative presented by the original authors on the paper Video Quality Assessment Based on Structural Distortion Measurement is to take a weighted average of SSIM results per channel on the YCrCb space like $SSIM_{final} = 0.8 \times SSIM_y + 0.1 \times SSIM_{Cr} + 0.1 \times SSIM_{Cb}$.