The reason to wash is cross contamination.
You are going to handle both the peeled and unpeeled.
The utensil (knife or peeler) will come in contact with both unpeeled and peeled.
A banana where you don't touch the peeled could be an exception. I sure there are other cases where you don't come in contact with the peeled.
Wash nuts would be a problem as they would get soft. I don't wash nuts.
Another reason to wash is to get rid of grit.
If you are to cut and only eat the inside (e.g. watermelon) you should should also wash for cross contamination.