The difference between the two is that the expression "a un passo da" can be used in a figurative way; I mean that "lui fosse a un passo da me" could also refer to the intellectual distance between the two people.
While the second one is always related to a physical distance; consider for example the expression "Facciamo due passi" (=let's go for a short walk).
For my experience, if you are referring to a physical distance the second form should be preferred and is a common way of saying. Said that, the first phrase could be used as well for the same purpose but its usage is less common.
To support my arguments, you can check these definitions at the dizionario dei modi di dire Hoepli (Hoepli dictionary of ways of saying):
a due passi
Vicino, a una distanza molto ridotta, che si potrebbe coprire in due passi. (Close, at a distance that can be covered in two steps)
fare un passo indietro
Fig.: regredire, perdere quanto si era precedentemente conquistato peggiorando la propria situazione. (Figurative, to step back, to loose what gained previously, worsening one own's situation)
Var.: essere un passo indietro
esserci un passo
Esserci pochissima differenza, riferito a due cose, situazioni o persone. (being a very little difference, referring to two things, situations or people)