When we have 'restrictive' or 'integrated' relative clauses we can usually use that instead of which or who(m). We don't usually use that with non-restrictive relative clauses:
- *Your Dad, that I've known for many years, is a really nice guy. (wrong)
There are some special cases when we can't use that with restrictive relative clauses either. The following sentence is fine:
- That's the man that you were talking with.
However, if we move the preposition to the front of the relative clause, that cannot be the complement of the preposition. We have to use whom or which as appropriate:
- That's the man with whom you were talking.
- *That's the man with that you were talking. (wrong)
[Notice that we cannot use who as the complement of a preposition. We have to use whom:
- *That's the man with who you were talking. (wrong)
Many writers on this site tell people not to use whom, but this is one situation where you have to!]
Another situation in which it's better not to use that, is after the demonstrative pronoun, 'pointy' that.
- #Do not do that that you know to be wrong. (awkward)
This is not grammatically wrong, but it's a bit awkward, and it can be difficult to read. Lastly we may want to contrast the difference between a person or thing that might have done something. In this case we may want to say who or which. This cannot work with the word that:
- Never trust in any people or in any things who, or which, you cannot actually see.
- *Never trust in any people or in any things who, or that, you cannot actually see. (wrong)
The Original Poster is, therefore, correct: we cannot always use that instead of who(m) or which. The following sentences from the original question are ungrammatical, because that cannot appear as the object of a preposition:
- *This is the man to that I spoke. (wrong)
- *This is the ball with that I scored. (wrong)
I hope this helps!