0

Let us consider the configuration of a 2D system and the standard definition of entropy $H=-\sum_{i=1}^{m}p_{i}\cdot \log(p_{i})$. Let us suppose that I can describe the state of my system by a 2D distribution over a square grid and suppose to consider two configurations (i.e. two distributions over these square grid ) which have the same entropy. I would like to know if there exist modified measures of entropy/information which take into account also clustering so that a clustered configuration of my system is no more degenerate with respect to a more sparse one.

user1234383
  • 123
  • Although it's not perfectly clear what you want, your request sounds remarkably like this question, which has answers: http://stats.stackexchange.com/questions/17109/measuring-entropy-information-patterns-of-a-2d-binary-matrix. – whuber Jun 11 '13 at 20:35
  • the problem is that I do not have enough reputation score to post an image since I am newcomer of this forum, otherwise it would be very simple to explain what I am looking for by this picture. – user1234383 Jun 11 '13 at 21:36
  • If you post a picture somewhere on the Web, you can supply a link to it. – whuber Jun 11 '13 at 21:41
  • link – user1234383 Jun 11 '13 at 21:48
  • I would like to have a measure according to which the two configurations are not degenerate and in particular a measure in which clustered configurations have a lower entropy than sparse ones as shown in the figure. – user1234383 Jun 11 '13 at 21:50
  • It still seems to me that is precisely the question addressed by the thread I linked to. If it is not, could you explain how they differ? – whuber Jun 11 '13 at 22:38
  • Thank you for the linked thread, but, if I understand your answer in that thread, I can modify the method you suggest by suitably defining the neighboring procedure to take into account clustering in my information measure. I have two problems with respect to this approach:
    1. in my systems too few occupied blocks to see make the sum on the neighbors for large k
    2. I would like a measure which include this without looking the system at different scale, something which a bonus in my entropy when specific structure are observed.
     – user1234383 Jun 12 '13 at 08:41
  • I understand the underlying reasons of the proposed method but I do not understand how in practice you reduce the number of blocks from 5x5, 4x4, 3x3, etc and which are the neighbors on which you are performing the sum. In the link you suggest for the neighboring procedure there is an explanation of that but I don't find any trace on how to reduce the size from 5x5, 4x4, 3x3, it seems to me that if I follow the simple example proposed I still obtain a grid of the original size. link you suggest – user1234383 Jun 12 '13 at 08:50

1 Answers1

1

Given that this measure (and all similar measures) reduce a complete data set to a single value, pretty much any not perfectly correlated second measure will help distinguishing such situations.

For example, you could rotate your coordinate system by 45° and then compute entropy on the rotated grid. Maybe one can construct a configuration that has the same entropy in the first measure, but not when rotated 45°.

+--+--+    +--+--+
|  |  |    |  | O|
|  |O |    |  |  |
+--+--+    +--+--+
| O|  |    |  |  |
|  |  |    |O |  |
+--+--+    +--+--+

Nearby objects may then end up in the same cell when using a different grid - or not:

X   X   X      X   X   X  
 \ / \ / \      \ / \O/ \ 
  X  OX   X      X   X   X
 / \O/ \ /      / \ / \ / 
X   X   X      X  OX   X  
 \ / \ / \      \ / \ / \

An even simpler example is just to vary the grid size!

+---+---+---+     +----+----+
|   |   |   |     |    |    |
|   |   |   |     |    |    |
+---+---+---+     |    |O   |
|   |  O|   |     +----+----+
|   |O  |   |     |   O|    |
+---+---+---+     |    |    |
|   |   |   |     |    |    |
|   |   |   |     +----+----+
+---+---+---+
Has QUIT--Anony-Mousse
  • 42,358
  • Thank you for your answer but entropy should be invariant under the rotation of the system that you propose and in general under permutation/rearrangement of the cells. $H(p1,p2,p3)= H(p2,p3,p1) = H(p2,p1,p3)$ and etc, even if I rotate the system the $p_i$ which enter in the sum defining the entropy are the same, therefore I would find the same result. – user1234383 Jun 12 '13 at 08:33
  • Your grid will change due to the rotation, and I'm not aware of a continuous notion of entropy. – Has QUIT--Anony-Mousse Jun 12 '13 at 08:50
  • Although it's not perfectly clear to me which rotation you are proposing, in my case given the fact that I don't have many observations for building the empirical distribution on which I calculate the entropy, I fear that the rotation will produce an almost equal situation, however I will try. Thank you. – user1234383 Jun 12 '13 at 09:01
  • why continuous notion of entropy? I don't get the point with this observation. – user1234383 Jun 12 '13 at 09:02
  • Grids cause artifacts. If you change the grid, your get different artifacts. – Has QUIT--Anony-Mousse Jun 12 '13 at 09:24
  • Ok, now I understand and thanks for the suggestion. Indeed I was trying to varying the size of the grid and I will try the rotation, now I get your point. I think that continuous limit of the entropy does not exist, at least does not exist trivially in fact for continuous distribution the generalization of the definition sum->integral result to be ill-defined. – user1234383 Jun 12 '13 at 10:58