You may want to have an abstract approach to this; we can exploit the fact that amsmath changes all operator names taking limits in a uniform way, by adding @ at the end of the name to mean the symbol.
\documentclass{article}
\usepackage{amsmath,etoolbox}
\makeatletter
\newtoggle{reducedop}
\newcommand{\reduceoperatorsize}[1]{%
\csletcs{#1saved}{#1}%
\csletcs{#1@saved}{#1@}%
\@namedef{#1}{%
\@ifstar
{\togglefalse{reducedop}\@nameuse{#1saved}}%
{\toggletrue{reducedop}\@nameuse{#1saved}}%
}
\@namedef{#1@}{%
\iftoggle{reducedop}
{\reduced@operator{#1}}
{\@nameuse{#1@saved}}
}%
}
\newcommand{\reduced@operator}[1]{%
\mathop{\mathpalette\reduced@operator@i{#1}\relax}%
}
\newcommand\reduced@operator@i[2]{%
\ifx#1\displaystyle\textstyle\else#1\fi
\csname #2@saved\endcsname
}
\makeatother
\reduceoperatorsize{sum}
\begin{document}
$\sum*_{k=1}^n$\quad$\sum_{k=1}^n$\quad$\sum\limits_{k=1}^n$
\[
\sum*_{k=1}^n\quad\sum_{k=1}^{n} k^2=\frac{n(n+1)(2n+1)}{6}
\]
\end{document}
The first line is meant to show that the usual behavior is respected.

The amsmath package redefines \sum with
\let\sum@\sum
\def\sum{\DOTSB\sum@\slimits@}
and does the same for \coprod, \bigvee, \bigwedge, \biguplus, \bigcap, \bigcup, \prod, \bigotimes, \bigoplus, \bigodot and \bigsqcup. This makes the symbols “dots aware” and respecting the sumlimits or nosumlimits option because of
\DeclareOption{sumlimits}{\let\slimits@\displaylimits}
\DeclareOption{nosumlimits}{\let\slimits@\nolimits}
Based on this, I redefine \sum to be
\togglefalse{reducedop}\sumsaved
if \sum* is found and
\toggletrue{reducedop}\sumsaved
otherwise. Here \sumsaved is the same as \sum as defined in amsmath. Then I redefine \sum@ to do \sum@saved if the toggle is set to false, where \sum@saved is the same as the original \sum@. If the toggle is set to true, instead, \mathop{\mathpalette\reduced@operator@i{sum}} is executed. Within \mathpalette I check the style: if it is \displaystyle, I use \textstyle, otherwise the current style and typeset \sum@saved. The surrounding \mathop is followed by \slimits@ as before.
\sumoperator as an indexed form of "choice" in process algebra, not for "numerical" formulae. – Bordaigorl Sep 27 '14 at 22:49\DeclareMathOperator*{\barr}{\textstyle\sum}to retain the smaller size. But limits will change position for display and text style. Is that OK? – Steven B. Segletes Sep 27 '14 at 22:53\Sigmawhat you look for? – yo' Sep 27 '14 at 22:53\DeclareMathOperatorgoing to set\textstylefor everything following\barras well? – Bordaigorl Sep 27 '14 at 23:38\mathrm{\Sigma}is not the same as\textstyle\sum! – Bordaigorl Sep 27 '14 at 23:40textstyle. See\barrin my answer http://tex.stackexchange.com/questions/116506/how-are-big-operators-defined/116538#116538 for comparison. If you add\frac{1}{2}after a use of\barrin display environment, it will not be in\textstyle. – Steven B. Segletes Sep 28 '14 at 00:06\mathord{\textstyle\sum}_{i=3}^6might be something that could be used, though the surrounding spaces are not quite correct... – Steven B. Segletes Sep 28 '14 at 01:10