Apparently, the nxlmi font has the wrong glyph at sizes 10 and 5, while it has the right one at size 7.
Until the issue is fixed at the font level, you can somewhat fix it by telling LaTeX to get the “g” from a different font.
\documentclass{article}
\usepackage{mathtools,amssymb}
\usepackage{libertine}
\usepackage[libertine]{newtxmath}
\DeclareSymbolFont{libertineg}{\encodingdefault}{\familydefault}{m}{it}
\SetSymbolFont{libertineg}{bold}{\encodingdefault}{\familydefault}{b}{it}
\DeclareMathSymbol{g}{\mathalpha}{libertineg}{`g}
\begin{document}
\[g \ \mu_g \ A_{\mu_g}\ \textit{g}\]
\end{document}

If you want the “variant g”, it's a bit more complicated: we need to scale down the text style “g” only when in script style.
\documentclass{article}
\usepackage{mathtools,amssymb}
\usepackage{libertine}
\usepackage[libertine]{newtxmath}
\usepackage{xfp,graphicx}
\newcommand{\fixg}{{%
\mathchoice
{\normalg}% displaystyle
{\normalg}% textstyle
{\scaledg}% scriptstyle
{\normalg}% scriptscriptstyle
}}
\makeatletter
\newcommand{\scaledg}{%
\scalebox{\fpeval{\sf@size/\f@size}}{$\m@th\normalg$}%
}
\begingroup\lccode`~=`g \lowercase{\endgroup\let~\fixg}
\edef\normalg{\mathchar\the\mathcode`g }
\AtBeginDocument{\mathcode`g="8000 }
\begin{document}
\[g \ \mu_g \ A_{\mu_g}\]
\end{document}

libertineoption is not active. Contact the maintainer ofnewtxmathand tell them about it (the project README on CTAN has an email address at the end). – moewe Dec 01 '19 at 07:10gin text, script, and scriptscript math style, I suggest you replace\usepackage[libertine]{newtxmath}with\usepackage{libertinust1math}. (Aside: Michael Shell is the author of both packages.) Note that the latter package does not offer avargoption at present; hence, no single-storeygin math mode. – Mico Dec 01 '19 at 08:22