If both of you are not interested in the event, just say so. Mention that you are not really into that kind of event, but would love to come if it were a "insert some event here". Try to find an event that both you and your agent has similar interest.
Oh, we are not a big fan of music concert, sorry. But I hear you are a big fan of "the Cats" (or whatever). Are you free next week? We would love to watch them together with you!
By inviting you, it is most likely that your agent might be trying to improve relationship with you. You might want to reconsider turning down the invitation for the relationship sake, if you really want to improve your relationship. However, do not feel obliged to accept, if you really want to keep him at your arm's length.
He might not putting too much hope if he's really just an acquaintance, and if he is offended by your decline, you really want to reconsider keeping relationship with him.