"not" pseudo-class is ignored

Question

I'm trying to understand why :not(...) doesn't work as expected. See the example:

.wrapper {
    background: #eee;
    padding: 0.1em 1em;
}

:not(.wrapper) a[href^='#'] {
    font-style: italic;
}

<p><a href="https://google.com/">Link to Google. Nothing special about it</a></p>
<p><a href="#foo">Link to some section in this document. It should be italic</a></p>
<p><a href="#bar">Link to some another section in this document. It should be italic</a></p>

<p>Special wrapper. Anchor-links inside it should NOT be italic. But they are:</p>

<div class="wrapper">
    <p><a href="#aaa">This link should NOT be italic</a></p>
    <p><a href="#bbb">This link should NOT be italic</a></p>
</div>

It seems that :not(.wrapper) part is ignored, i.e.

:not(.wrapper) a[href^='#'] {
    font-style: italic;
}

reads as

a[href^='#'] {
    font-style: italic;
}

Is there a way to fix or workaround it?