Shogakukan's 大国語辞典 shows that 姪【めい】 has a historical hiragana spelling of めひ, not めい, showing that the modern mei reading is not on'yomi but rather kun'yomi. This different derivation is probably also why the pronunciation is different: [mei] with a more distinct [i], and not [meː]. The term for "nephew", 甥【おい】, has a historical hiragana spelling of をひ. As you note, め is an old morpheme indicating "female", and を is an old morpheme indicating "male". We now have a clear follow-on morpheme ひ.
This ひ is of uncertain derivation. The closest vaguely relevant match that I can currently find is prefix 曾【ひ】, often glossed as "great" in terms like 曾【ひ】お祖母【ばあ】さん "great-grandmother", 曾【ひ】孫【まご】 "great-grandchild", etc. Shogakukan's given definition for 曾【ひ】 is:
血縁関係を表す語について、それよりさらに一代離れた関係にあることを示す。
For terms expressing a blood relation, this indicates a relation separated by one more generation.
I wonder if there might have been an older meaning, still referring to relative distance from oneself, but not necessarily of a different generation; and an older usage, not as a prefix but as a noun. I'll keep looking.
ひこ【彦】 \
(「日子」の意) \
---- \
ひ【日】 \
... \
7 皇室や皇族に関する事柄につけて、ほめたたえる気持を表す語。日の神、すなわち、天照大神の子孫の意とも、光り輝く太陽にたとえたことばともいう。「日の御子」「日の御門」など。
– Eiríkr Útlendi Nov 07 '15 at 00:55