Yes, to me that usage is fine. In that context, "would have been" is substantially identical to "was," except that "would have been" typically is meant to convey a connotation that the speaker is trying to remember/has to think about when it was, as opposed to knowing definitively when it was (here, also indicated by speaker's use of "The only time I believe...").
In a different context, "would have been" of course means that what comes after is speculative/uncertain, and a trace of that is present in the connotation. A more express statement of this concept would be "I don't believe I've met him, but if I have it would have been at your mother-in-law's 70th birthday dinner."
Incidentally, I find the phrase "physical pleasantries" strange. Pleasantries are relatively content-free phrases like "It is nice to meet you," "How are you," &c. These are also described as "phatic," I am informed by another post. Anyway, since "pleasantries" are usually verbal (but may be written), "exchanged physical pleasantries" sounds to me like a euphemism for some type of casual sexual activity. I suppose that a handshake or brief hug could be said to be "exchanging physical pleasantries," but I have never heard this phrase used before.