2

In a previous question (Entropy of an image) and in various sources on the web, the Shannon entropy of an image is considered to be the entropy of the frequency distribution of the grayscale values.

In this way a random image where the pixels take the values 0 or 1 with the same probability has an entropy equal to a single coin toss: we have 50% white and 50% black so the entropy is 1. This does not vary with the dimension of the image.

enter image description here

What I am not able to understand is why each pixel is not considered as a separate random variable.

In my opinion, in my example, is more correct to consider an image with $n$ pixels as a sequence of $n$ coin tosses, resulting finally in an entropy of $n$:

$$H(I)= -2^n 2^{-n} \log_2 2^{-n} = n$$

In the case of an 8-bit image (256 possible pixel values), the final result would be $8*n$. This measure of entropy is dependent on the number of pixels on the image. Something that makes sense to me. Moreover, it solves the problem presented in the previously linked question (Entropy of an image) of the spatial correlation of the pixels.

enter image description here

In this case, the number of free pixels would be decreased because each line is completely dependent on the value of its first pixel. This would decrease the entropy.

What am I doing wrong?

User1865345
  • 8,202
Edoardo
  • 21
  • 1
    Shannon entropy is defined for discrete random variables in general. There is no 'the' entropy of an image since you can define a variety of discrete random variables pertaining to images that will all have their own entropy. Of course you can have separate pixel-wise entropies! – Galen Jun 26 '23 at 14:37

2 Answers2

2

There is no such a thing as the "entropy of an image". Entropy is about the random variable. If you want to be talking about entropy, you need to think of the image in terms of a random variable(s). There are many ways how you can think of an image as a random variable. For example, it can be a random variable with two states "image contains cat" and "image does not contain a cat" with associated probabilities. Another way could be to consider the image recorded as an $m \times n$ random matrix (assuming black-and-white, for colors you would have additional dimensions) where each pixel is a separate random variable. For white noise images, the random variables would be independent and identically distributed, so their joint distribution would be $p(x_1, x_2, \dots, x_{m\times n}) = \prod_i p(x_i)$, but for any other images, the pixels would be non-independent and not necessarily identically distributed. For any of the random variables (univariate, multivariate), you can calculate the entropy using the regular formula.

Tim
  • 138,066
1

A single image doesn't have the property of information entropy.

It is the probability distribution of images that has the property of information entropy.

When the entropy of a single image is considered, then it can be about a probability distribution of images that are conditioned on having the same properties as that single image.

This is related to physics where one considers macrostates and microstates. The latter, a microstate, has no entropy in itself, but in a way you could consider the entropy of the related macrostate and some model for it's distribution.

So when you consider the entropy of an image, and consider all $k^n$ states equally possible, then an image is indeed just like a discrete uniform distribution with entropy $\Omega = \log_2 k^n = n \log_2 k$.

It is not wrong to consider the entropy of an image like that. However, one may consider also other uses of entropy (that are more useful).

For example, when one considers images according to a model that entails a non-uniform distribution, where certain values are more/less likely, then one can have lossless image compression by encoding the more likely patterns with fewer bits (at the cost of the less likely patterns requiring more bits), such that on average less bits are needed to encode the image.

Sextus Empiricus
  • 77,915