Here's a solution that needs at most
5
weighings.
For convenience, I arbitrarily number each stone on side "A" (which I declare to be the lighter side for convenience) of the balance in the first weighing to be stones 1, 3, 5...19 and each stone on the heaver side B 2, 4, 6...20. The original weighing of all of the stones is taken to count toward the weighing total, but will be named "step zero" for convenience, because it's the only possible thing to do at the beginning.
Note: if at any point the scale balances, a not-lightest stone can be found in one more weighing by switching any two stones and seeing which way the balance tips.
Step one:
Swap stones 1, 3, 5, and 7 with stones 2, 4, 6, and 8. If the balance tips, you would skip to step 3, but that wouldn't be worst-case scenario (as you will see), so we won't skip ahead. We assume the balance does not tip. Total weighings: 2
Step two:
Swap stones 9, 11, 13, and 15 with stones 10, 12, 14, and 16. If the balance tips doesn't tip, you would skip to step 4, but that's not the worst case, so we will assume that the balance does tip. Total weighings: 3
Step three:
Swap four of the 8 stones that you have just swapped. Assuming that we are still following the worst case, swap stones 9 and 11 with 10 and 12. If the scale does not tip back to side B, then you know that switching 13 and 15 with 14 and 16 will. Total weighings: 4
Step four:
Regardless of which route we took to get here, we currently have four stones which we know either have tipped the scale when swapped, or would tip the scale when swapped. Swap two of them across the balance. If the scale doesn't change state when these are swapped, then the unswapped stone on the side that the scale was expected to tip away from is definitely not the lightest stone. If the scale does change state, the swapped stone that ended up on the side that the scale just tipped toward is definitely _not the lightest stone. Total weighings: 5
And there's your answer!