After an intellect devourer reduces a target's Intelligence score to 0, it can attempt to consume the target's brain with a contested check.
When this scenario came up during a recent Adventurers League game, I gave the affected player a -6 modifier on the contested check. That was a DM judgment call. I've verified that abilities with a 1 score have a -5 ability modifier in the PHB, but 0 is not on the chart.
We know every even number is a + or - depending on which way above or below the base of 10 we go, so it would look like this:
\begin{array}{|c|c|} \text{Score} & \hline \text{Modifier} \\ \hline 20 &+5 \\\hline 18-19&+4\\\hline 16-17&+3\\\hline 14-15&+2\\\hline 12-13&+1\\\hline 10-11&0\\\hline 8-9&-1\\\hline 6-7&-2\\\hline 4-5&-3\\\hline 2-3&-4\\\hline 0-1&-5\\\hline \end{array}
Note that -6 does not happen unless the score somehow drops to a number below 0. If we follow the chart lower, then -1 or -2 would give a -6 modifier, -3 or -4 would give a -7 modifier, and so forth. Same with scores above 21. 22 and 23 grant a +6 modifier. Your judgement call was in error, but that's why it's a DM judgement call. Sometimes the answers do not readily present themselves. No harm done, and now you know better.
This can be seen in the text immediately following the chart you reference, at the top of the next column (PHB p. 173):
To determine an ability modifier without consulting the table, subtract 10 from the ability score and then divide the total by 2 (round down).
In your case, (0−10)/2 = -5.
