Is the general word order of this sentence correct?
We investigate how strong the effect of X on Y is.
Or, as an alternative,
We investigate how strong the effect of X is on Y.
In a preprint for a scientific publication I came across the formulation:
We investigate how strong is the effect of XZY.
and now I'm all confused.
The Object clause how strong ..., since it starts with a Wh-word, is an Embedded Question Object Complement clause. Embedded Wh-Questions differ from real Wh-Questions in that they don't undergo Subject-Auxiliary Inversion.
So that's what lets out the third example, which has undergone Subject-Auxiliary Inversion.
Both of the first two examples are fine, because they both come from different potential placements of the prepositional phrase on Y. Being a locative adverbial, it can be inserted into any available niche in the sentence, including
Both of these, if "Z" is a real variable, coreferential with What, produce
as a non-embedded Wh-Question. However, as an Embedded question, either order is available, since the auxiliary verb is doesn't get inverted with the subject Noun Phrase the effect.
Summary:
While I personally think the second example is stylistically strongest, there's nothing really wrong with the first. It just makes the reader parse "X on Y" as a prepositional phrase before getting to the copula, and "the effect of X" is easier to digest as a unit than "the effect of X on Y."
Moreover, although the third example may have an odd feel to it, there is still nothing wrong with that word order either. I can imagine the formulation being used to emphasize the word strength. (I'm not sure what XZY is supposed to mean; did you intend to write "X on Y" there as well?)
Imagine an expressive voice reading the line:
We investigate how strong is the effect of X on Y.
And consider a parallel construction:
We wonder how foolish is the man who thinks otherwise.
A little bit poetic, perhaps a bit dramatic — but certainly no glaring solecism to be worried about.
Addendum
On the surface, this obviously conflicts with John Lawler's view about the third example. He is answering as a grammarian and for most uses his view on that example would be considered standard. I only mentioned my point about No. 3 because English is really more flexible than "standard grammar" would allow.
The "poetic" or "dramatic" effect I refer to is called anastrophe. It refers to an inversion of word order used for effect as a rhetorical device. From Ward Farnsworth's Classical English Rhetoric:
Some standard purposes of [anastrophe]:
a. The unexpected placement of words calls attention to them. Pushing a word into an especially early or late position often creates emphasis in itself; then the emphasis is still greater because the ordering mildly violates the reader's expectations.
<p>b. Inversion may put words in an order that creates an attractive rhythm.</p> <p>c. Inversion may compress a meaning into fewer words.</p> <p>d. Inversion sometimes causes the full meaning of a sentence to become clear only late in its progress; this bit of suspense makes the finish more climactic when it arrives</p>
That said, in most cases it would be safer to avoid anastrophe if you don't really know how to use it or intend to do so — to use it, as @AndrewLeach suggests, you first need to understand that you are deviating from a standard or "normal" word order.
