Say there is a system with open-loop transfer function $$ G(s)=\frac{1}{(s-1)}$$. The system is definitely unstable.
If I put any nonlinear system in cascade with $G(s)$ and from the output, a negative unity feedback. Is it possible to make the system close-loop transfer function is stable? It is assumed that the higher order harmonics are filtered in the close loop path.
What you are suggesting is approached in the area of adaptive control systems, and yes it is possible to achieve asymptotic closed loop stability of your example, and other similar systems by using adaptive control methods.
Like the suggestion made by @user31748, adaptive control will effectively cancel the unstable pole and provide stabilizing dynamics, but unlike the direct cancellation method, the adaptive control can adjust for perturbations, if they exist in the unstable pole position.
That's where the flaw exists in direct cancelling of unstable poles - the unstable dynamics typically move around in real life.
If you are not familiar with adaptive control, I recommend a tutorial book, Adaptive Control Tutorial, written by my former advisor at USC, Petros Ioannou and one of his grad students, Baris Fidan. Petros is a leading expert in adaptive controls, and the book is a good balance between theory and practice.
