At the moment I'm a bit baffled. What sentence part is "visitors" in "I'm not allowed visitors"? I would not call it an object or a subject complement as "to be allowed" is no linking verb.
The only idea I have is to say the sentence is elliptic and the full form would be "I'm not allowed to have visitors". Here "visitors" is object of the to-infinitive.
So is it possible to say "visitors" is object of a to-infinitive (to have) that was deleted?