Yes, it's rather a convention. Changes of word order convey shades of meaning and are used for emphasis of certain elements within a sentence.
The direct order of мой друг makes the phrase sound descriptive like in a sentence Он - мой друг - He's a friend of mine.
In contrast the order of друг мой makes the phrase sound vocative, i.e. expressing an address - Hey, my friend! (NOT Hey, a friend of mine!).
Привет, мой друг is not incorrect, but less vocative than the other version.
That said, if the adjectival cluster related to the noun друг (or any noun denoting an addressee) is more elaborate, the noun itself does tend to come last, e.g. Привет, мой юный закадычный друг!
The word order in Привет, друг мой could also be an expression of a disguised reproach, disapproval, disappointment.