Is there a difference between the variants? Are they all correct to use?
Your second and third sentences are fine, but the parenthesized alternatives with ever before had are klunky; I'd stick with putting ever after had. And sentences in this form, a declarative clause with an 'embedded question', are not ordinarily pointed with a question mark.†
I was wondering if they had ever seen each other before.
I was wondering whether they had ever seen each other before.
Your first sentence, however, is not acceptable in formal registers as you have pointed it. I think you may be confused by two very different uses of if.
The inversion Had they ever seen ... is equivalent to If they had ever seen ... only in conditional constructions, where if introduces the condition clause (protasis). In conditional constructions the condition clause is a clausal adjunct, restricting the condition under which the action or state asserted in the main clause is true. If the condition clause is deleted you will be left with a syntactically complete main clause:
[If they had ever seen each other] they would have quarreled.
[ Had they ever seen each other] they would have quarreled.
In your sentences, however, where if acts as an equivalent to whether, the clause which if introduces is not a conditional adjunct but a complement to the main-clause verb; if that clause is deleted you will be left with a syntactically incomplete clause:
I was wondering [if they had ever seen each other].
A 'complementizer',‡ if or whether, is obligatory here.
However, the inverted form had they ever seen is a syntactically well-formed question; and with a somewhat different pointing you could make your first sentence entirely acceptable:
I was wondering—had they ever seen each other before?
I should probably add that the use of the wh- words and if is complicated. These words introduce clauses of several types—frank questions, bound relative clauses, fused relative clauses, and embedded questions—and although all these types have the same underlying 'interrogative' semantics and the same underlying 'gapped' mechanic, they differ in details. I haven't yet seen a theoretical treatment which makes the relationship between the different types clear.
† This is not to say that your use of a question mark is wrong—it is quite common and entirely proper to employ the question mark with a declarative sentence if you intend your reader to understand that it is spoken with interrogative inflection. And of course a sentence like this is only going to appear in a relatively informal context, where you are emulating the spoken dialect. You should be aware, however, that in formal registers we avoid writing sentences whose meaning is dependent on a particular intonation, because this puts a greater burden of interpretation on the reader.
‡ I put 'complementizer' in quotes because this is not how the term is ordinarily used; but signalling that what follows is a complement is in fact one of the syntactic functions of wh- words and if in this sort of context.