I couldn't come = I was unable to come
You couldn't have seen me = It's not possible you saw me (since I wasn't there).
He couldn't have killed her. He was with me all night.
"A firework couldn’t have done all that damage."
http://dictionary.cambridge.org/grammar/british-grammar/could
A sentence with 'couldn't have + p.p.' proves a previously made statement wrong. A sentence with 'couldn't + inf.' simply states a fact; it isn't proving anything wrong. The construction 'couldn't have +p.p.' cannot stand alone. It needs to be preceded by a false statement, whereupon the construction is used, followed by proof, to prove the statement is wrong.
For example: false statement : This little boy wrote the essay in front of you. Response: He couldn't have written it; he hasn't even learned the alphabet yet./ He couldn't write his homework today because he broke his wrist.
In your example, you state a fact; you don't prove anything wrong. You say 'I couldn't come to the party; I had to work.
It was not possible = I was unable = I couldn't come , but I couldn't have come = it's impossible for you to conclude that I was present, for example, You say you saw me at the party, but I couldn't have been at the party since I was at home. It's awkward to say: I couldn't have come to the party. - though it has the same meaning as I couldn't have been at the party - You couldn't have seen me there. (not I had another obligation - that's I couldn't come)