The matrix contains a description to move form one space to another. The implied assumption is that there is some parent space which the matrix is defined in. There can be arbitrary chains of these parent child relationships. So you need to know the parent of the matrix.
The matrix multiplications $M \cdot \vec{v}$ describes the vector $\vec{v}$ in child space $M$ as seen by the parent space. The inverse $M^{-1} \cdot \vec{v}$ does the reverse of this it describes the vector in parent space in terms of child space coordinates.
So simply if you make a tree out of your parent child relationships when you move towards the root of your tree (World) you use non-inverted matrices, and when you move in the opposite direction (towards a leaf/child node) you take the inverse.
Why would you need the inverse? Well because you want to move two things independently of each other but still use the results in one space. For example imagine a world with one object and a camera. Its efficient to be able to model the camera separately. But then the vectors need to move form object space to camera space. So you take the object space move to world (moving towards root non inverted multiplication) and then you move that to camera space (move towards a child of world use inverse) so you get:
$$T_{camera}^{-1} \cdot (T_{object} \cdot \vec{v}) = T_{camera}^{-1} \cdot T_{object} \cdot \vec{v}$$
(Please note that matrix multiplication order depends on whether you define vector as row or column matrices. Both are fine mathematically as is storing the sub vectors in arbitrary order inside the matrix)