The claim of a first cause wouldn't be that it was always there, it would be that at some point there was nothing, and then the first cause made it so that there suddenly started being something. Or perhaps the universe was in some perfectly static state, and then the first cause caused stuff to start happening (although then one would also need to posit that the universe was in that initial state).
But if there is nothing (or if things were perfectly static), there would be nothing to be a first cause, so that seems to be a not-insignificant problem.
This leads to some odd ideas, like something causing itself, or something causing an event in the past (that ultimately causes itself). I've heard hypotheses of such things in quantum mechanics, although I'm not too well-versed in that and I don't know how popular such hypotheses are.
Also, with the spacetime physics model, it's at least theoretically conceivable (if not most likely) that time and how we think of causation (something always having a cause, always being caused by things that happened before it, and only being able to cause things after it) breaks down at the earliest points of the universe. At that point, it may be rather difficult (if not nonsensical) to try to discuss what was "first".
The much more common way that a first cause is posited is by calling it "timeless".
In theory, this sidesteps the problem by just asserting that ideas such as "before" and "always" doesn't apply to the cause.
In my view, this assertion is unjustified and contrived in apologetics (it's asserted as a property of God, so God can serve as the only solution to a presented problem, rather than being a conclusion reached based on any sort of evidence or rational justification). It also introduces a far bigger problem of what it even means to be timeless, how something could exist timelessly, and how a timeless entity could have the properties attributed to it (e.g. consciousness).
Yes, I did just allude to how we think of causation breaking down in the section above, which could conceivably allow for the possibility of timelessness. But the breakdown of causation is a counter-intuitive conclusion (not assumption) that would apply to elementary physics particles. If you want the first cause to have consciousness, this introduces additional problem of how it's possible to think without time (given that we tend to consciously think fairly sequentially) and space (given that the only consciousnesses we know of are tied to physical bodies*). Also, if timelessness, and causation from it, is possible in physics, God would not be necessary to terminate the regress, which rather undermines e.g. people trying to prove God's necessity using the cosmological argument.
* Idealism posits non-physical consciousness, and that reality is a result of consciousness. This renders the entire discussion of causes within reality somewhat moot. It might solve the first cause of observed reality (that cause then trivially being consciousness), but it mostly just pushes back the problem, and creates a few more, because then the question would become what caused that consciousness, or what was the "first cause" within that consciousness, and where and how does that consciousness exist, and how is that consciousness linked to observed reality, and what explanatory power or justification does the claim of such a consciousness have.
