The borrow rules of Rust need to be checked at compilation time, that is why something like mutably borrowing a part of a Vec is a very hard problem to solve (if not impossible), and why it is not possible with Rust.
Thus, when you do something like &mut v[i], it will mutably borrow the entire vector.
Imagine I did something like
let guard = something(&mut v[i]);
do_something_else(&mut v[j]);
guard.do_job();
Here, I create an object guard that internally stores a mutable reference to v[i], and will do something with it when I call do_job().
In the meantime, I did something that changed v[j]. guard holds a mutable reference that is supposed to guarantee nothing else can modify v[i]. In this case, all is good, as long as i is different from j; if the two values are equal it is a huge violation of the borrow rules.
As the compiler cannot guarantee that i != j, it is thus forbidden.
This was a simple example, but similar cases are legions, and are why such access mutably borrows the whole container. Plus the fact that the compiler actually does not know enough about the internals of Vec to ensure that this operation is safe even if i != j.
In your precise case, you can have a look at the swap(..) method available on Vec that does the swap you are manually implementing.
On a more generic case, you'll probably need an other container. Possibilities are wrapping all the values of your Vec into a type with interior mutability, such as Cell or RefCell, or even using a completely different container, as @llogiq suggested in his answer with par-vec.