I tried to find out what you might mean, but unfortunately you do not provide an MWE. Even worse, my crystal ball got stolen. So I can only guess what you mean. I guess you are confusing frame number with page number. To see this consider the MWE
\documentclass{beamer}
\setbeamertemplate{footline}[page number]{}
%\setbeamertemplate{footline}[frame number]{}
\begin{document}
\begin{frame}[noframenumbering]{First frame}
\end{frame}
\begin{frame}[t]{Second frame}
\end{frame}
\begin{frame}[t]{Third frame}
\end{frame}
\begin{frame}[noframenumbering]{Fouth frame}
\end{frame}
\begin{frame}[t]{Fifth frame}
\end{frame}
\end{document}

This seems to confirm what you're saying, but the catch is that I am showing the page number here. Indeed, if I activate \setbeamertemplate{footline}[frame number]{} instead, I get

Is this the phenomenon you're referring to?