Units are often understood to be part of a random variable. If someone writes that "$h_i$ is the height of the $i$th patient ", you're meant to assume that every $i$ uses the same unit and it's a sensible one given the context (say, meters if $i$ indexes over adult humans). This is often omitted for the sake of brevity, but it would clearer if people explicitly wrote out things like "$h_i$ is the height of the $i$th patient in meters."
Units restrict how you can manipulate variables (and constants, too!). You can only directly add or subtract variables with the same units, which yields a result with the same units as the original. Variables with different units can be multiplied or divided, with the result having compound units. If $X$ is in meters and $Y$ in seconds, $\frac{X}{Y}$ has units of $\frac{\text{m}}{\text{s}}$ and $XY$ has units of $\text{m}\cdot \text{s}$. You can also multiply and divide by unitless values, like the number of data points, which does not change the result's units. Thus, the mean has the same units as the data from which is calculated, as does the standard deviation. Variance, however, has squared units: if $X$ is in meters, $V(X)$ has units of $\text{m}^2$.
Some statistical operations remove the units. Suppose you have a set of heights $h_i$ measured in meters. If you plug these into into the formula for $Z$-scores
$$z_i = \frac{h_i - \bar{h}}{\sigma_h}$$
you'll notice that the units cancel:
$\frac{\left(\text{meter} - \text{meter}\right) \overset{\Delta}{=} \text{meter}}{\text{meter}} = 1$, so $z_i$ is unitless. In a sense, this is the whole point: the $Z$ score is used to make variables, potentially measured with on different scales and with different units, comparable. Covariance and correlation also make a useful pair of examples. If you grind though the formula, you'll find the the covariance of $X$ and $Y$ has units of $x \cdot y$ and a scale that depends on the original variable. Correlation includes a "normalization" term that rescales this to between -1 and +1, while removing the units. This too is intentional: it lets you use correlation to compare relationships between disparate quantities.
Others find conversions between units. You can interpret regression as finding the "best" way to convert between the dependent and independent variables. Suppose you want to crudely predict people's heights. Using a set of heights (in meters), weights (in kilograms), and sex, you might fit a model like:
$$ \text{height} = \beta_0 + \beta_1 \text{weight} + \beta_2 \text{sex}$$
The coefficient $\beta_1$ needs to have units of $\frac{\text{m}}{\text{kg}}$ for the equation to balance, and that's exactly how you should interpret it. Holding everything else at its baseline level, a $1 \text{ kg}$ increase in weight corresponds to a $\beta_1 \text{ m}$ increase in predicted height! Categorical variables are conceptually similar. We usually encode them as (unitless) numbers, such as male=0, female=1. $\beta_2$ is therefore in units of meters. You can also imagine that it's in the fictitious units of meters per sex-code or something if you want to move the units through the encoding process instead.
This holds for linear models, but you can apply similar logic to other families. A logistic regression, for example, predicts probabilities and finds coefficients that can be interpreted as mapping between the predictors and changes in log odds.
Want to learn more? If you're familiar with dimensional analysis from a natural science class, you may enjoy this article Nick Cox found.
Finney, D. J. (1977). Dimensions of Statistics. Journal of the Royal Statistical Society. Series C (Applied Statistics), 26(3), 285–289. https://doi.org/10.2307/2346969
If you want more practice with dimensional analysis, check out the first chapter of Sanjoy Mahajan's Street Fighting Mathematics, available with open-access here.
