If we can say: He is considered (to be) smart. We consider him (to be) smart.
By the same token, I have also heard: We think him (to be) smart or think him dead.
It is also correct to say: He is said to be smart.
But, Is it correct to say???: We say him to be smart.
For some reason it appears to follow the above structures, but sounds very strange.
In standard English usage, the verb say can have as its object either a literal utterance, a noun phrase, or a subordinate clause. But even in the last two cases, the object must refer to at least a figurative utterance. Four examples:
1. He glanced up and said, "No thank you." [an utterance]
2. She lowered her head and said grace. [a noun phrase]
3. He says that he'll do it. [a subordinate clause]
4. Her facial expression said she was fed up.
In example 4 the object is a subordinate clause (with the conjunction that left out). And the utterance is only figurative: Her facial expression conveyed her exasperation as though it could speak, and if so it would say, "I'm fed up."
When it comes, as in your example, to expressing the opinion that someone is smart, the closest might be things like
We declare him to be smart.
or
We proclaim him to be smart.
But each of those has at least a suggestion of a speech act
In speech, the most common way to express this is:
We say he is smart.
"We say him to be smart." is not idiomatic English.
Question re: "We think him (to be) smart or think him dead." is OK but it is somewhat literary, and somewhat BrE. It generally would not be used in contemporary AmE speech. But it is found in novels and older publications.
Think [someone] [adjective] is OK.
Generally, though, smart in British English means well dressed or well turned out.
