Actually the 蒜 is evidence of a substrate influence from an Austroasiatic language (ancestor of Vietnamese xoài or Khmer ស្វាយ [svaay]) in this context. Indeed, this lexeme actually does have its own Chinese character, 檨, with Mandarin pronunciation shē (ㄕㄜ).
It is the Hakka and Min varieties which use this 檨 lexeme:
- Taiwanese Min Nan: soāiⁿ (Peh-oe-ji), suāinn (Tai-lo), ㄙㄨㆮ˫ (Zhuyin)
- Fuzhounese Min Dong: suông (Bang-ua-ce)
- Taiwanese Hakka: sôn (Phak-fa-sü)
In many of these varieties, the pronunications of 蒜 are very close:
- Taiwanese Min Nan: soàn (Peh-oe-ji)
- Fuzhounese Min Dong: sáung (Bang-ua-ce)
- Taiwanese Hakka: son (Phak-fa-sü)
Hence the substitution of 蒜 was motivated by phonetic similarity (假借) with the lack of "availability" of this 檨 character.
In many of these, the collocation 番檨, preferred in Min Dong and certain varieties of Hakka, may help distinguish it from just 蒜. Or it may just be that mangoes came a bit later to the northeastern Fujian area than the southern area.