With credit to @ConsigliereZARF for helpful comments and references. The earliest definition of group character ("Charakter") for Abelian groups is likely due to Weber (1881-2), and it was generalized to general groups by Frobenius (1896). According to Mackey's survey Harmonic analysis as the exploitation of symmetry:
"In 1881 Weber defined a character of a finite commutative group $G$ to be a complex valued function $\chi$ on $G$ such that $\chi(xy) = \chi(x)\chi(y)$ for all $x$ and $y$ in $G$. This definition was an abstract generalization of one given three years earlier by Dedekind in connection with his work on algebraic number theory, which was inspired in turn by early work of Gauss and Dirichlet (see sections 6 and 12). While Weber's definition makes sense for arbitrary finite groups, it is more or less vacuous except insofar as the group has commutative aspects. Specifically, every character is identically one on the commutator
subgroup and consequently the only characters not identically one are
derived trivially from characters of commutative quotient groups.
Group
theory acquired a powerful new tool that was soon to become almost indispensable
when G. Frobenius (1849-1917) published a paper in 1896 showing
that there is a natural generalization of the character notion that involves
the whole group $G$ in a significant and interesting way—even when $G$ is
non-commutative."
Mackey appears to refer to Weber's Beweis des Satzes, dass jede eigentlich primitive quadratische Form unendlich viele Primzahlen darzustellen fähig ist (1882), where he generalizes Dirichlet's prime number theorem. Weber introduces the character using a basis of Abelian group and remarks that it always satisfies the multiplicative property. In a footnote he credits Dedekind's Supplement XI to the third edition of Dirichlet's Vorlesungen über Zahlentheorie (1879), p. 581.
Dirichlet uses the word "Charaktere" throughout Vorlesungen über Zahlentheorie, the earliest on p.77 in connection with quadratic reciprocity, and then expands the use of the name (p. 316). Dirichlet's characters are not quite group characters, and the notion of a group was not in place at the time of Gauss's or Dirichlet's writing. However, concerning Gauss's "total character" of a quadratic form from Disquisitiones Arithmeticae (1801), Mackey writes:
"The word character as used today stems directly from Gauss's use of the
term in his theory of binary quadratic forms... Dedekind makes the same definition for the special case of the ideal class group
of an algebraic number field. As Dedekind himself had pointed out earlier,
the ideal class group is a generalization of Gauss's group of equivalence
classes of binary quadratic forms. Dedekind's mode of expression is such as
to make it clear that he regards his definition as a generalization of that of
Gauss."
Before Gauss, quadratic characters, later generalized by Jacobi and Dirichlet, appear in Legendre's Essai sur la theorie des nombres (1798). Introducing the quadratic character symbol, he simply wrote:
"As the quantities analogous to $N^{\frac{c-1}2}$ will frequently appear in the course of our research, we will use the abbreviated character $\left(\frac{N}{c}\right)$ to express the residue from dividing $N^{\frac{c-1}2}$ by $c$; the residue which, according to what we have just seen, can only be $+1$ or $-1$."[ See Cajori, History of Mathematical Notation, p.30]
So to him "character" meant nothing more than "symbol". Whether this is where Gauss took the word from is unclear, Mackey says that large parts of Disquisitiones Arithmeticae were written before Legendre's Essai was published, but Gauss "acknowledges that he was inspired to study quadratic forms by the work of Lagrange and Legendre". Legendre divided classes of forms into cosets that Gauss called genera, and forms were said to be in the same genus if they had the same "total character", a system of $+1$s and $-1$s canonically associated to them. Each $\pm1$ was produced by a function on forms what we today call a group character.
