After installing May 2018 CU for SharePoint 2016 (16.0.4690.1000), both Authentication Providers (Windows & custom STS) lead to the custom STS one. Before CU installation, when selecting Windows authentication, it showed the Windows credentials popup, as it obviously still should. It's working fine in PROD environment that hasn't yet been updated (version 16.0.4573.1000), but not in this test farm.
Basically https://sharepoint.company.com/_windows/default.aspx?ReturnUrl=XYZ&Source=XYZ redirects to the custom authentication provider instead of showing Windows credentials prompt. Issue occurs on all browsers even in privacy mode.
I've ran IISRESET, tried toggling the Auth Providers for the web application to perhaps refresh something somewhere, but nothing makes the Windows authentication selection work like it should. web.config is practically identical to what it was - just few additional assembly references due to new features in this CU (FP2).
I've also compared the Microsoft.SharePoint.IdentityModel.dll, and related web.config and ASPX pages, and while the assembly has been updated, the essential parts AFAIK haven't been changed in any way.
If I disable the custom authentication provider from the Authentication Providers for the web application, Windows authentication prompt comes in just fine.
Any tips what I could try to make it work?