When we say mustn't be, we mean cannot be. If we want to say that they either could or could not be, we will say don't have to be or needn't be or don't need to be.
You don't need to wear formal dress.
You musn't wear formal dress.
The first one means that formal dress is optional; the second means that formal dress is forbidden.
So, to your examples. Can't/couldn't/mustn't have been lovers are basically equivalent.
I saw Peter and Jane holding hands. Since they were both married, they couldn't/can't/mustn't have been lovers.
These all mean roughly the same thing. Mustn't usually conveys a bit less forcefulness than can't or couldn't, but it is saying the same thing. For example:
He can't have been wearing a red tie. He hates red ties.
He must not have gone downtown this afternoon, since I saw him sitting on his porch.
The first is drawing a little bit stronger conclusion than the second, but they are both saying that the condition described is sufficient evidence to draw the conclusion.
Now, consider this:
Peter and Jane don't have to be lovers just because I saw them holding hands.
This is saying that the fact of holding hands isn't enough evidence to draw the conclusion that they are lovers. However, they might be.
Now, the simple past mustn't have a past participle. That means that you can not use a past participle with the simple past.
This, by the way, is the opposite of German. "Es muss nicht sein, aber es kann sein" translates "It doesn't have to be, but it can be." "It must not be" translates "es darf nicht sein."