Margin paragraph alignment if using \sloppy

Question

Please see my MWE. I would like the \marginnote paragraph to be aligned with the point in which I invoke it. In the inner margin I have a wrap figure environment that contains text very big (46pt). If I don't use \sloppy, text of margin paragraph and of main paragraph overlap. If I use \sloppy, I run into problems with alignment. Please, what am I doing wrong? Thanks.

\documentclass[10pt]{book}
\usepackage{lipsum}
\usepackage{marginnote}
 \renewcommand*{\marginfont}{\scriptsize \sffamily}
   \renewcommand\raggedrightmarginnote{}
\renewcommand\raggedleftmarginnote{}
\usepackage[
paperwidth=160mm, paperheight=240mm,
textheight=18cm,
textwidth=8cm,
inner=2cm, marginparwidth=2.5cm, marginparsep=0.5cm]
    {geometry}
\usepackage{wrapfig}
\usepackage{fontspec}
\begin{document}
%\sloppy
\newpage
\paragraph{First}
\begin{wrapfigure}{i}{0.64\textwidth}%
  \vspace{5pt}
  \noindent  {\fontsize{46pt}{46pt}\selectfont \hspace{-5pt}  Big}
  \end{wrapfigure}
This\marginnote{I want this to be aligned. \lipsum[1]} is a paragraph. \lipsum[1]
\newpage
\paragraph{First}
\begin{wrapfigure}{i}{0.64\textwidth}%
  \vspace{5pt}
  \noindent  {\fontsize{46pt}{46pt}\selectfont \hspace{-5pt}  Big}
  \end{wrapfigure}
This\marginnote{I want this to be aligned. \lipsum[1]} is a paragraph. \lipsum[1]
\newpage
\paragraph{Forth}
\begin{wrapfigure}{i}{0.64\textwidth}%
  \vspace{5pt}
  \noindent  {\fontsize{46pt}{46pt}\selectfont \hspace{-5pt}  Big}
  \end{wrapfigure}
This\marginnote{I want this to be aligned. \lipsum[1]} is a paragraph. \lipsum[1]
\end{document}