When TeX finds ' (in math mode), it basically does the following:
- it emits
^\bgroup\prime;
- it looks at the following token (without expanding it);
- if the next token is another
' it emits \prime and returns to step 2;
- if the next token is
^, it gobbles it and absorbs what comes next as a macro argument (either a single token or a braced group) and considers it as part of the exponent that's being built and emits \egroup;
- otherwise it emits
\egroup.
In your case, the next token is \inv so this falls into case 5. Thus
C'\inv
becomes
C^\bgroup\prime\egroup\inv
and then
C^\bgroup\prime\egroup^{-1}
Double superscript.
If you want to support this syntax you have to add to the lookup also \inv.
All considered, I'd use the syntax ^\inv instead.
Anyway, here's an expl3 implementation that supports \inv.
\documentclass{article}
\ExplSyntaxOn
\cs_new_protected:Nn \wang_prime:
{
% start a superscript
\c_math_superscript_token
\c_group_begin_token
% we want a prime
\prime
__wang_prime_next:
}
\cs_new_protected:Nn __wang_prime_next:
{
\peek_charcode_remove_ignore_spaces:NTF '
{ % another ' follows; add it and restart the recursion
\prime
__wang_prime_next:
}
{ % look for ^ or \inv
\token_if_eq_meaning:NNTF \l_peek_token \c_math_superscript_token
{ % remove ^ and add the superscript
__wang_prime_remove_superscript:Nn
}
{
\token_if_eq_meaning:NNTF \l_peek_token \inv
{
__wang_prime_remove_inv:N
}
{ % finish the business
\c_group_end_token
}
}
}
}
\cs_new_protected:Nn __wang_prime_remove_superscript:Nn { #2 \c_group_end_token }
\cs_new_protected:Nn __wang_prime_remove_inv:N { -1 \c_group_end_token }
\char_set_active_eq:NN ' \wang_prime:
\ExplSyntaxOff
\newcommand{\inv}{^{-1}}
\begin{document}
$A^{-1}$
$B\inv$
$C'^{-1}$
$D''^{-1}$
$E'\inv$
\end{document}
Note that a'_{1}^{2} is not supported and cannot be.

$C^{\prime-1}$. – barbara beeton May 30 '22 at 20:08