Fontspec: Misplaced hacek above i when loading TeX Gyre Pagella with fontspec

Question

When I load TeX Gyre Pagella using fontspec's setmainfont command and try to use a hacek above an i, it is misplaced.

The reason I am using fontspec is because I want old style numbers to be the default, and I can set that with fontspec's setmainfont.

If I instead load TeX Gyre Pagella using fontenc and tgpagella, the hacek is aligned properly.

However, if I do this, I have to manually specify that I want to use old style numbers using \oldstylenums{}, which is a real pain (including specifying them in things like enumerated lists, page numbers, etc., which requires manually tweaking a bunch of things I'd rather not have to mess with).

Is there a way to load the correct glyph while still using fontspec? Or else some way to manually adjust it to look right?

MWE:

\documentclass{article}
%\usepackage[T1]{fontenc}
%\usepackage{tgpagella} % These work for \v{\i}, but no default old style numbers
\usepackage{fontspec}
\setmainfont[Numbers=OldStyle]{TeX Gyre Pagella} % Default old style numbers, but bad \v{\i}
\begin{document}
    \v{\i}
\end{document}

Edit: As Thérèse's comment makes apparent, it might be relevant that I'm using XeLaTeX.