When I use a mixer (Behringer 4-track), the instructions say to wait about ~30 seconds after turning off the phantom power before unplugging anything. They say not doing that could damage the equipment.
Now I have an XLR-to-USB connection that provides phantom power (48V, like the mixer), and there's nothing about that in the instructions. Is it safe to unplug the mike right away?