Since beamer version 3.08, there is an undocumented option noframenumbering that doesn't increase the slide counter for the current frame. So if you want to exclude a single frame from the numeration and the handout, you can use
\begin{frame}<handout:0>[noframenumbering]
...
\end{frame}
To automate this, i. e. to exclude all the frame that aren't displayed in the current mode (handout, trans) from the numeration, you can patch the internal beamer command responsible for generating the frames:
\usepackage{etoolbox}
\makeatletter
\pretocmd{\beamer@@@@frame}{\alt<#1>{}{\beamer@noframenumberingtrue}}{}{}
\makeatother
Include this code somewhere in the preamble (that's between \documentclass{beamer} and \begin{document}) of your document. It uses the overlay command \alt to activate noframenumbering on every slide which isn't displayed.
Example code:
\documentclass[handout,gray]{beamer}
\usepackage[utf8]{inputenc}
\usepackage{pgfpages}
\pgfpagesuselayout{4 on 1}[a4paper,border shrink=5mm,landscape]
\setbeamertemplate{headline}{\scriptsize{\vspace*{0.3cm}\hspace*{0.3cm}\insertframenumber}}
% https://tex.stackexchange.com/a/49806/3323
\usepackage{etoolbox}
\makeatletter
\pretocmd{\beamer@@@@frame}{\alt<#1>{}{\beamer@noframenumberingtrue}}{}{}
\makeatother
\begin{document}
\begin{frame}
Absolute frame number: 1\\
Frame number in presentation: \insertframenumber
\end{frame}
\begin{frame}
Absolute frame number: 2\\
Frame number in presentation: \insertframenumber
\end{frame}
\begin{frame}<handout:0>
Absolute frame number: 3\\
Frame number in presentation: \insertframenumber
\end{frame}
\begin{frame}
Absolute frame number: 4\\
Frame number in presentation: \insertframenumber
\end{frame}
\end{document}
Result (click on the image to see it full-size):

\alt<display>{4}{3}to put a4when indisplaymode and3otherwise (iehandoutmode, thougharticlewould also get the3). Would this help, or is the real case more complicated? – Andrew Stacey Mar 26 '12 at 19:57