The convention has no enforcement mechanism of any kind, states can (and do) easily ignore it and many also have pretty funky reservations (as an example with some relevance today, note that Turkey stipulated it only intends to apply the 1951 convention to European refugees).
So the convention only serves as a kind of self-commitment device and simply denouncing it would not be very different from ignoring it, it makes the breach of commitment visible and you can expect criticism either way. Which critics would be placated by such a move?
Beyond that, while the Convention includes many detailed procedural rules, it (and other related treaties) can also be regarded as a way to codify and reaffirm principles that are widely taken as self-evident (cf. the notion of jus cogens). So denouncing it could in principle give states more leeway but not completely free them from all obligations.