When it seems a Pope has heretical beliefs, who determines whether those beliefs are Magisterial?

Question

I know that even some Catholics think that certains Popes held personal heresies (were material heretics). For example Honorius I believed that Jesus had only one will, the heresy of monothelitism.

I understand that a Pope might be a heretic, but to preserve the believed infallibility of his office he can't impose those beliefs ex cathedra—i.e., using the full weight of his office—onto the Church.

Who has the authority to judge which statements issued by the Pope are infallible?

Note: This question seems to address what the "criteria" used are, but I am asking: Who uses these criteria to decide the matter?

Answer 1

Whether valid popes have been heretics

It isn't necessarily true that Pope Honorius I was a heretic. St. Robert Cardinal Bellarmine, the great doctor of the papacy, defended several popes, including Pope Honorius I, against accusations of heresy in his work that is translated into English as Papal Error?: A Defense of Popes said to have Erred in Fatih (ch. 4 is on Pope Honorius I). Among the seven accusations against Pope Honorius I that St. Robert's refutes are the the accusations that Dialogist quotes below.

St. Robert also treats the question of whether a Pope can be a heretic in De Romano Pontifice, lib. II, cap. 30 (On the Roman Pontiff). He argues that a pope cannot be a heretic; if he is, he, like St. Francis de Sales wrote in his Catholic Controversy p. 306,

falls ipso facto from his dignity and out of the Church, and the Church must either deprive him, or, as some say, declare him deprived, of his Apostolic See, and must say as S. Peter did: Let another take his bishopric.*
^{*Acts i.}

Who can judge a pope's validity?

As I mentioned in my answer to your other question, only a valid Pope is the supreme judge:

Can. 1442 The Roman Pontiff is the supreme judge for the entire Catholic world; he renders judicial decisions personally, through the ordinary tribunals of the Apostolic See, or through judges he has delegated.

This law is based upon the infallible dogma of papal primacy which, pace Dialogist's answer above, has been true since the time Christ instituted the Papacy (Matt. 16:16-19), founding it on the rock of St. Peter himself, the first Pope. As the First Vatican Council (Pastor æternus ch. 4 "Of the Institution of the Apostolic Primacy in blessed Peter"), united with the Pope, infallibly declared:

We therefore teach and declare that, according to the testimony of the Gospel, the primacy of jurisdiction over the universal Church of God was immediately and directly promised and given to blessed Peter the Apostle by Christ the Lord. For it was to Simon alone, to whom he had already said: "Thou shalt be called Cephas," [John i. 42.] that the Lord after the confession made by him, saying: 'Thou art the Christ, the Son of the living God,' addressed these solemn words: 'Blessed art thou, Simon Bar-Jona, because flesh and blood have not revealed it to thee, but my Father who is in heaven.

And I say to thee that thou art Peter; and upon this rock I will build my Church, and the gates of hell shall not prevail against it. And I will give to thee the keys of the kingdom of heaven. And whatsoever thou shalt bind on earth, it shall be bound also in heaven; and whatsoever thou shalt loose on earth, it shall be loosed also in heaven.' [Matt. xvi. 16-19.] And it was upon Simon alone that Jesus after his resurrection bestowed the jurisdiction of chief pastor and ruler over all his fold in the words: 'Feed my lambs; feed my sheep.' [John xxi. 15-17.] At open variance with this clear doctrine of Holy Scripture as it has been ever understood by the Catholic Church are the perverse opinions of those who, while they distort the form of government established by Christ the Lord in his Church, deny that Peter in his single person, preferably to all the other Apostles, whether taken separately or together, was endowed by Christ with a true and proper primacy of jurisdiction; or of those who assert that the same primacy was not bestowed immediately and directly upon blessed Peter himself, but upon the Church, and through the Church on Peter as her minister.

If any one, therefore, shall say that blessed Peter the Apostle was not appointed the Prince of all the Apostles and the visible Head of the whole Church Militant; or that the same directly and immediately received from the same our Lord Jesus Christ a primacy of honor only, and not of true and proper jurisdiction: let him be anathema.

Any Catholic can and must recognize and refuse to follow an antipope.

However, any Catholic who recognizes an antipope as an antipope must refuse to follow him. St. Paul says twice in Galatian 1:8-9:

But although we, or an Angel from heaven, evangelize to you beside that which we have evangelized to you, be he anathema.

As we have said before, so now I say again, If any evangelize to you beside that which we have evangelized to you, be he anathema.

Notice, St. Paul says anyone, which certainly includes those who appear to be popes (antipopes).

Answer 2

Pope Honorius I served from 625 to 638 when the Church was still undivided. At that time there was no doctrine within the Church of Papal infallibility and the Pope's place with respect to the other Patriarchs was understood to be first in honor, but he had no authority over any bishop outside the jurisdiction of the Roman See. These things were innovations which led to or occurred sometime after the See of Rome separated mutually from the other four ancient Sees (Constantinople, Alexandria, Antioch, Jerusalem) in the 11th century.

(I would hasten to add that although the Pope had no jurisdiction outside of the Roman See, Patriarchs and the faithful would frequently appeal outside their Patriarchate for spiritual guidance and support when problems appeared from within. This was the case, for example, during the iconoclast heresy when Church Fathers such as Germanus and John of Damascus appealed for support from Pope Gregory II against Emperor Leo the Isaurian. The case of Honorius shows that the system also worked in the opposite direction: the faithful of Rome had recourse to the eastern Patriarchs.)

Pope Honorius was condemned and anathematized by the council of bishops which met during the Sixth Ecumenical Council at Constantinople in 680-681. His sentence reads:

The holy council said: After we had reconsidered, according to our promise which we had made to your highness, the doctrinal letters of Sergius, at one time patriarch of this royal god-protected city to Cyrus, who was then bishop of Phasis and to Honorius some time Pope of Old Rome, as well as the letter of the latter to the same Sergius, we find that these documents are quite foreign to the apostolic dogmas, to the declarations of the holy Councils, and to all the accepted Fathers, and that they follow the false teachings of the heretics; therefore we entirely reject them, and execrate them as hurtful to the soul. But the names of those men whose doctrines we execrate must also be thrust forth from the holy Church of God

The declaration which anathematized Honorius:

Many years to the Emperor! Many years to Constantine, our great Emperor! Many years to the Orthodox King! Many years to our Emperor that maketh peace! Many years to Constantine, a second Martian! Many years to Constantine, a new Theodosius! Many years to Constantine, a new Justinian! Many years to the keeper of the orthodox faith! O Lord preserve the foundation of the Churches! O Lord preserve the keeper of the faith!

Many years to Agatho, Pope of Rome! Many years to George, Patriarch of Constantinople! Many years to Theophanus, Patriarch of Antioch! Many years to the orthodox council! Many years to the orthodox Senate!

To Theodore of Pharan, the heretic, anathema!
To Sergius, the heretic, anathema!
To Cyrus, the heretic, anathema!
To Honorius, the heretic, anathema!
To Pyrthus, the heretic, anathema!
To Paul
To Peter
To Macarius
To Stephen
To Polychronius
To Apergius of Perga the heretic, anathema!

To all heretics, anathema! To all who side with heretics, anathema!

May the faith of the Christians increase, and long years to the orthodox and Ecumenical Council!

These documents can be found in the English translations available of the Acts and Canons of the Ecumenical Councils (e.g. CCEL)