“I’ve to go,” and similar sentences, are correct English, but they’re not idiomatic American English. Searching in Ngrams shows that “I’ve to go,” turns up primarily in novels, spoken by characters who are supposed to have a pronounced dialect. “I’ve to leave,” on the other hand, seems to be regional, fairly common in books by authors with South Asian names.
I’ve only found one example of a non-fiction book using the exact phrase you’re asking about, “I’ve to work,” and it’s a self-help book whose grammar seems a little off to me. (Manifestation: Manifest Your Dream Life Full of Happiness by Michael Tolle and Eckhart Losier.)
Contracting the helper verb have is much more common in other contexts, especially the perfect tense. The Google Books corpus shows that “I’ve gone” slowly began catching up to “I have gone” around 1900 and is now more common, but “I have to go” is more than three hundred times as common as “I’ve to go.” From other comments, contracting have as a main verb is common in British but not American English.
So, I don’t know of any logical explanation. It’s just something that Americans never say.