In short, $X$ is a random variable. It's not concrete value. It's like a representation of a random event. And yes, $E[X]$ is expected value of all observations of $X$.
If we talk about $E[X] = \frac{1}{n} \sum_{i=1}^{n} x_i$ - that's sample mean (or arithmetic mean). Sample mean is not "real" mean. It's like an approximation by the known value $\{x_1, \dots, x_n\} \in X$.
For example, for a continuous distribution of $X$ - $E[X] = \int_{-\infty}^{\infty} x f(x) dx$, where $f(x)$ is the probability density function.