Firstly, in this context, "got" = "have", even though the tenses dont match. The idea is, if you got something (past tense) OR if you've gotten (present perfect) something, you now have it.
This not only works for possesion
"I've got 3 dollars" = "I got 3 dollars" = "I have 3 dollars",
but also posession
"I've got to go" = "I got to go" = "I have to go"
Regardless of the actual tense, these three mean you must go now. The phrase implies you must go due to social obligation, physical need, etc.
"I had to go" is the past tense version. e. g. "I had to go to school on Saturday when I was a young boy".
"I've had to go" is the present perfect tense version. e.g. "I've had to go to the bathroom for 2 hours now (and still have to)".
On the other hand, "I get to go" is a compeletly different expression that means "I am permitted (either by someone or the circumstance) to go). e.g. "I get to go to Disneyland next week".
The past tense version of this is "I got to go". "I got to go Disneyland last week".
As you may have noticed "I got to go" can be used in both expressions. You have to figure out which one through context.