For one thing, "papolatry" was a word invented by anti-catholics not more than 200 years ago, and does not reflect actual Catholic doctrine — unsurprisingly. That is to say, it substitutes a straw man for genuine Catholic doctrine, in order to make it look bad, because the real doctrine doesn't look bad enough — because the real doctrine doesn't demand 'papolatry.' The straw man is implying Catholic doctrine demands that Catholics worship popes, or that everything popes say or do is considered to be infallible and irrevocable.
Since this isn't Catholic doctrine, that is the response or defense: we don't have to defend what we don't believe.
However, there is an actual error among some Catholics ignorant of Catholic doctrine, which is that they will defend everything a pope says as though Catholic doctrine were that everything the pope says or does is infallible or orthodox, which it does not.
Today, if you criticize the last few popes for literally hosting idolatry meetings or kissing the Qur'an, which blasphemes Christianity, you are labelled a heretic by such people. And it's implied that since they cannot be wrong, they were ignorant or naive — but not sinning or giving any harmful impression to the faithful.
This isn't 'papolatry' so much as it's just misunderstanding the doctrine of papal infallibility, which is much more restricted in scope than these people realize.
Suffice it to say, the Church has never taught that popes must be believed absolutely, in every word or action, and thus the defense against this error is simply rejecting the premise of the straw man in the first place.