Use incentives, not punishment
There are more effective ways to keep players being murderous cretins than retribution and punishment. In my experience, players generally don't enjoy losing or being beat up, while they do enjoy being appreciated.
Give them social relationships that matter. Reward good behaviour, with gold, services, gratitude of the populace, friends, and successful outcomes of plans. Have not every ally or encounter betray them eventually, so they don't learn to better kill first. And so on. Giving them a reason to not be murder hobos, by showing them it leads to good outcomes is more effective and more fun than putting them down with superior force.
Retribution measures
But, you ask about retribution, so let us address this. It would not make sense to list specific features, because powerful NPCs can be built like PCs, and can have access to literally anything a PC could have access to, wether these are magic items, class features, or spells. DMG p. 92:
When you give an NPC game statistics, you have three main options: giving the NPC only the few statistics it needs, give the NPC a monster stat block, or give the NPC a class and levels.
In addition, they can be allied with creatures and monsters of any CR, from the monster manual and otherwise. And, as you tagged your qeustion homebrew, NPCs are of course not limited to what is available to player characters, they can have other powers, tools, and features that you as the DM deem appropriate.
Depending on the world, the most powerful NPCs are of much higher level than the PCs for most of the PCs career. Most play happens in tier one and two, while powerful NPC spellcasters that have been published, or even NPCs from the monster manual have access to spells up to 9th level, meaning they have spellcasting abilities comparable to a 17th level character.
In very practical terms, if your PCs want make it all the way up to level 17, they need to learn not to piss off those powerful NPCs along the way.
Magical medieval society
When it comes to society, nobody is an island. Even the little hamlet that has no means to defend themselves from a level 7 party that wants to throw their weight around is part of someones fiefdom, and that someone will not be amused. If the PCs cause trouble enough, and for long enough, what they do will eventually trickle up to someone who is powerful enough to put a stop to it.
The basic idea hear is that society protects itself by escalation. If they cannot deal with the PCs with local forces, like a group of city guards, the escalate to more powerful defenses, and in a magical fantasy world, such powerful defenders exist.
Imprisonment. You seem to assume one answer to such misbehaving PCs is imprisonment by bounty hunters, and from looking at your list of measures, seem concerned on how to contain spellcasters. There is a whole question on that. The simplest, mundane way is to gag and tightly bind them, so they cannot use verbal or somatic componetns, and strip them of all foci and material components.
Execution. You also can have stricter laws, that punish murder with being put to death yourself. Dead PCs don't cast spells. I however would be careful -- it is easy to do that, but the question is, do you want to do that?