Why shouldn't old font commands such as \rm or \it or \bf used? Because they follow the original LaTeX setup and reset all font features. So if you do \it\bf you just get boldface and not italic.
The command \DeclareOldFontCommand should only be used for compatibility with older documents. If you look in the LaTeX kernel, you'll see that \bfseries is defined with \DeclareRobustCommand.
You have also to declare what bold font you plan to use, if you define semibold.
\documentclass{article}
\usepackage{fontspec}
\setmainfont{Linux Libertine O}[
BoldFont=* Bold,
BoldItalicFont=* Bold Italic,
FontFace = {sb}{n}{* Semibold},
FontFace = {sb}{it}{* Semibold Italic},
]
\DeclareRobustCommand{\sbseries}{\fontseries{sb}\selectfont}
\DeclareTextFontCommand{\textsb}{\sbseries}
\begin{document}
Hello world. \textsb{Hello world.} \textbf{Hello world.}
\textit{Hello world.} \textit{\textsb{Hello world.}} \textit{\textbf{Hello world.}}
\textsc{Hello world.} \textsb{\textsc{Hello world.}} \textbf{\textsc{Hello world.}}
\textit{\textsc{Hello world.}}
\textit{\textsb{\textsc{Hello world.}}}
\textit{\textbf{\textsc{Hello world.}}}
\end{document}
The fonts have been changed to suit my machine.

If I replace all periods with .\typeout{\fontname\font}, the terminal shows
"Linux Libertine O/OT:script=latn;language=dflt;mapping=tex-text;"
"Linux Libertine O Semibold/OT:script=latn;language=dflt;mapping=tex-text;"
"Linux Libertine O Bold/OT:script=latn;language=dflt;mapping=tex-text;"
"Linux Libertine O Italic/OT:script=latn;language=dflt;mapping=tex-text;"
"Linux Libertine O Semibold Italic/OT:script=latn;language=dflt;mapping=tex-text;"
"Linux Libertine O Bold Italic/OT:script=latn;language=dflt;mapping=tex-text;"
"Linux Libertine O/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"Linux Libertine O Semibold/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"Linux Libertine O Bold/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"Linux Libertine O Italic/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"Linux Libertine O Semibold Italic/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"Linux Libertine O Bold Italic/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
which seems to be what you're looking for.
Here's the setup for Libertinus Serif.
\documentclass{article}
\usepackage{fontspec}
\setmainfont{LibertinusSerif}[
Extension=.otf,
UprightFont=-Regular,
ItalicFont=-Italic,
BoldFont=-Bold,
BoldItalicFont=-BoldItalic,
FontFace = {sb}{n}{-Semibold},
FontFace = {sb}{it}{-SemiboldItalic},
]
\DeclareRobustCommand{\sbseries}{\fontseries{sb}\selectfont}
\DeclareTextFontCommand{\textsb}{\sbseries}
\begin{document}
Hello world\typeout{\fontname\font}.
\textsb{Hello world\typeout{\fontname\font}.}
\textbf{Hello world\typeout{\fontname\font}.}
\textit{Hello world\typeout{\fontname\font}.}
\textit{\textsb{Hello world\typeout{\fontname\font}.}}
\textit{\textbf{Hello world\typeout{\fontname\font}.}}
\textsc{Hello world\typeout{\fontname\font}.}
\textsb{\textsc{Hello world\typeout{\fontname\font}.}}
\textbf{\textsc{Hello world\typeout{\fontname\font}.}}
\textit{\textsc{Hello world\typeout{\fontname\font}.}}
\textit{\textsb{\textsc{Hello world\typeout{\fontname\font}.}}}
\textit{\textbf{\textsc{Hello world\typeout{\fontname\font}.}}}
\end{document}

Just to check what fonts are being used, I added \typeout{\fontname\font} and on the terminal I get
"[LibertinusSerif-Regular.otf]/OT:script=latn;language=dflt;mapping=tex-text;"
"[LibertinusSerif-Semibold.otf]/OT:script=latn;language=dflt;mapping=tex-text;"
"[LibertinusSerif-Bold.otf]/OT:script=latn;language=dflt;mapping=tex-text;"
"[LibertinusSerif-Italic.otf]/OT:script=latn;language=dflt;mapping=tex-text;"
"[LibertinusSerif-SemiboldItalic.otf]/OT:script=latn;language=dflt;mapping=tex-text;"
"[LibertinusSerif-BoldItalic.otf]/OT:script=latn;language=dflt;mapping=tex-text;"
"[LibertinusSerif-Regular.otf]/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"[LibertinusSerif-Semibold.otf]/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"[LibertinusSerif-Bold.otf]/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"[LibertinusSerif-Italic.otf]/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"[LibertinusSerif-SemiboldItalic.otf]/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
"[LibertinusSerif-BoldItalic.otf]/OT:script=latn;language=dflt;+smcp;mapping=tex-text;"
After seeing the output, both in the Linux Libertine and Libertinus serif cases, I believe that using both bold and semibold does not work, because the weights are not distinguishable enough: only if they're next to each other one can spot some difference. You should choose one of them as the boldface font and don't rely on both.
\DeclareRobustCommand{\sbseries}{\fontseries{sb}\selectfont}? – egreg Apr 11 '21 at 08:55LinLibertine_RZI_G.ttfdoes not have small caps. You can check this withotfinfo -f. – Davislor Apr 11 '21 at 13:10