A theory of observational errors first appeared in Simpson's memoir of 1755, which postulated that positive and negative errors are equally probable, and discussed several possible distributions of error over the interval where they were supposed to fall, including uniform and triangular distribution. In 1765 Lambert added the semicircle distribution to the list, and Lagrange added parabolic in 1776 and raised cosine and logarithmic distributions 1781. The first systematic attempt to derive the "error curve" was undertaken by Laplace in 1774, but the distribution he obtained had a cusp at the origin. He derived the "correct" error curve, the normal one, four years later, Gauss rediscovered it around 1795. The name is due to Pierce (1873), who studied measurement errors for an object dropped on a wooden base.
Tobias Mayer, in his study of the libration of the moon (1750), suggested to reduce observational errors by averaging over groups of similar equations instead of just averaging observations under identical circumstances. In 1755 Boškovic in his work on the shape of the earth proposed that the true value of a series of observations would be that which minimizes the sum of absolute errors, i.e. what we call the median. The method of least squares to minimize measurement errors was published independently by Adrien-Marie Legendre (1805), Robert Adrain (1808), and Carl Friedrich Gauss (1809). Gauss earned European fame in 1801 by using it to predict the location of Ceres based on rather imprecise data, see History of statistics.
Thus, some basic statistical tools were developed by the beginning of the 19-th century and tried in astronomy. But after that the major developments and applications of error theory happened not in physics but in "soft" sciences like medicine, psychology and sociology. In particular, the notions of standard deviation, correlation and regression were only introduced by Galton (1889) in field studies of human anatomy. Galton's work inspired Pearson to promote statistics as a basic tool of science, but it only became standard practice after Fisher's textbooks Statistical Methods for Research Workers (1925) and The Design of Experiments (1935), see Design of experiments.
Slow adoption of statistical methods in physics may have had something to do with the controversial status of statistical mechanics in the second half of 19th century. Bernoulli's 1738 idea of treating gases as large conglomerates of particles was picked up in 1859 by Maxwell, who derived the distribution of molecular velocities, and starting in 1870-s by Boltzmann, who became the greatest advocate of statistical methods in mechanics. But molecular theory was bitterly opposed by physical and chemical establishment (although, due to the success of the periodic table, chemists were much more receptive to the idea than physicists), led by Mach and Ostwald, as a metaphysical speculation, see Did anybody question the indivisibility of the atom because there were “too many” elements? Only after the success of Lorentz's theory of electrons, and especially Einstein's and Smoluchowski's theory of Brownian motion, which vindicated Boltzmann's ideas, was the barrier removed, see Statistical mechanics: history.