hi everybody!
The question is very controversial, truly. But there are some elements on which we can construct an answer. Firts I would like to start with a quote of the french author Chateaubriand, who writes in Mémoires d'outre-tombe "Macte animo generose puer" ascribing it to Virgil (here the Wikisource reference 1 with some notes). This false attribution shows however that this common say is well-known. If we try to analyse more deeply the uses in the latin litterature (my source is the Brepolis Research basis) we find that during the Antique period (<200) the common way of saying is used 15 times. Here the occurences :
- Calpurnius Flaccus Declamationum Excertpa 3, 3, 8
- Cicero ad att. 12, 6, 1
- Cicero, Tusc. 1, 17, 40
- Quintus Curtius Rufus Hist. Al. 4, 1, 18
- Horatius, Serm. 1, 2, 31
- Titus Livius 2, 12, 14
- Idem 4, 14, 7
- Idem 7, 10, 4
- Idem 7, 36, 5
- Idem 10, 40, 11
- Idem 22, 49, 9
- Idem 23, 15, 14
- Petronius, Sat. 94, 1
- Seneca ep. 66, 50
- S. Turpilius Comoediarum palliatarum fragmenta, v. 7
Hence, if we make an analysis of the construction of this adjective we can easily understand (from his initial meaning "honored, glorified") that "virtute" is an abative of limitation (because it limits acutally the field in wich someone has to be honored). Then, if the use of an adjective in the vocative with "esse" may be disturbing, we have to interpret it as the right case that match with an imperative (esto implies that a 2nd person exists, and the right case to refer to that person would be the vocative because in giving an order one is soliciting the semantic field of advocation). Then, the O. Riemann Syntaxe 7nth edition, Paris 1927 on page 170 §78 Note 2 says "With no doubt the vocative of an adjective who, at the classical time, seems to be indeclinable : "macte virtute...este" T. Livius 7, 36, 5 ; "iuberem macte virtute esse" Ibid 2, 12, 14.
Sorry if my english it may not be the best ever, but I'm an Italian scholar living and working in Paris.