First, you should be aware of the output of the two date difference should be in Days or Months or Years as a Number!
If you set The data type returned from this formula as DateTime, it would work but it logically would show invalid data.

So the difference should be shown as Number Not as Date format!
To get the diff between today and overdue date through this calculated column formula
=DATEDIF([overdue date],TODAY(),"d")

Note:
The calculated column formula is only calculated/updated in the
following cases:
- Add New Item.
- Update Existing item.
- Update the calculated column itself in the List Setting!
Check Also
To update the calculated column daily
Try to Create a Retention Policy to run a workflow which updates the list items daily!!
Or instead of using calculated column, use JSlink as mentioned at Days Past Since List Item Created - SharePoint Online 2013