Clothing items with 2 leg holes are generally plural. Clothing items with 2 arm holes aren’t.
Examples with 2 leg holes: pants, trousers, trunks, knickers.
Examples with 2 arm holes: shirt, pullover, jersey, jacket, vest, waistcoat.
Exceptions: in South Africa, we speak of a swimming costume (trunks in UK – still plural) or speedo. In the UK, usage is speedos, also plural (in Australia, budgie smugglers, note also plural).
Items with one leg hole are not plural: skirt, kilt. Items the are separately made for each appendage don’t count (gloves, shoes, socks): they logically should be plural and can be singular, if you only refer to one of them. Did anyone ever have a pant, a trouser or a trunk (elephants don’t count)?
So: why this distinction? Were leg coverings originally made of discrete items and only later became a single garment?