Typically if an attack roll equals the AC of a target then the attack does normal damage. I quite like the idea of making it a glancing blow, dealing half damage instead, this would help clue the PCs into what the AC of the target roughly is.
My only worry would be that this might create balance issues. I will make sure that the rule is the same for when monsters attack, so it is a general rule "ties do half damage." What would the consequences of this house-rule be?
I go by a rule of thumb that ~things~ hit/land/happen about two-thirds of the time in 5e. Really tough encounters you may make contact less than half the time, easy ones you're making contact four-in-five hits.*
So let's assume for argument's sake that "contact"--successful attack or failed save--happens 65% of the time, and does an average of X damage. Your expected damage per attack under the existing rules is then $$0.65 \times X$$
Your house rule takes one of those results and halves the damage: expected damage is now $$0.6 \times X + 0.05 \times (X/2)$$
or $$0.625 \times X$$
That difference--0.025 X--is minimal. If your attacks are doing 40 damage per hit (high tier 3 or even 4 by this point, or consuming considerable resources to bump it up) it's a difference of 1hp.
But if you're worried about "balance," you can just make the half-damage apply both at AC and AC-1. Then you've not changed the long-term numbers at all though you have slightly bumped the occurrence of hits, in case you're worried about on-hit effects.
* I'm explicitly ignoring extreme edge cases (like a 5% chance to hit or miss). If you're designing encounters like that you should already be comfortable with the probabilities implicated, in my opinion.
I personally use this as a mechanic, on both sides, having the players give and receive glancing blows.
I don't think its unbalanced as long as you make everyone (players and opponents) play by the same rules, especially since the chance of having an attack match the AC is less likely than either a hit or a miss.
Try it out, you might find it doesn't ever happen.
As a side note I enjoy the added flavour it gives to combat and, as you said, can give hints to enemy AC as well as how their attacks fair against the PC's AC.
Because the damage being dealt is slightly lower than normal, combat would go slower with this type of house rule.
Which is great if that's something you wish to see happen - combat is a big draw for some players. But it does affect a few other game mechanics too...
It also means they'll know exactly how high a roll they need to make if they hit the monster's AC - which isn't a huge advantage in a fight, but something they might reasonably be able to figure out in actual combat (how hard it is to hit something, that is).
Classes that are proficient in dealing high damage will have a slight disadvantage under these rules - they'll be dealing less damage on average and thus will be slightly less effective at their job.
Characters and monsters with high AC will be taking slightly less damage on average, and thus won't go down quite as quickly - they'll be slightly better at their job.
There are already some mechanics that half damage - off the top of my head, Barbarians take half damage to common damage sources when under Rage - so you might have to half the damage twice for a hit that hits AC - in addition to spells which might also do half-damage (and Barbarians who have the Bear Totem class feature) - which complicates things a bit, but not too severely.
Under this house rule, damage will be done slightly slower and battle will take slightly longer, but high AC characters will feel rewarded for their high AC, so it might be a net benefit.
It also might make some mechanics more difficult to manage, but not impossible and only in rare occasions would it come into play.
So overall it sounds like a good rule for making it feel a bit better for players who just barely missed having their armor deflect a blow, but might slow combat down a little.
Minimal effect. It makes combat a bit more survivable, by making attacks that hit slightly less harmful; damage will drop by about 2.5%. This will sometimes frustrate your players but they'll benefit from it more than they'll lose, for an overall (minor) PC power increase.
The universal approach is to ...
...instead of just doing an average case and flying with that.
That is, what happens if you need 20, 19 or 2 to score a hit?
In this case, the easy way is to study the average Joe Fighter with a longsword and str 16 that deals 1d8 + 3, or 7.5 points of damage per hit. Let's ignore crits for simplicity...
Without the rule
To hit: 20 – Average damage is \$ 7.5 \cdot \frac{1}{20} = 0.375\$
To hit: 19 – Average damage is \$ 7.5 \cdot \frac{2}{20} = 0.750\$
To hit: 2 – Average damage is \$ 7.5 \cdot \frac{19}{20} = 7.125\$
With the rule
To hit: 20 – Average damage is \$ 7.5 \cdot \frac{1}{20} \cdot \frac{1}{2} = 0.1875\$ or 0.3750 depending on interpretation/crit immunity/etc.. In any case, the reduced value would be 50 % of old damage.
To hit: 19 – Average damage is \$ 7.5 \cdot (\frac{1}{20} + \frac{1}{2}\cdot\frac{1}{20}) = 0.5625\$ (or 75% of old damage)
To hit: 2 – Average damage is \$ 7.5 \cdot (\frac{18}{20} + \frac{1}{2}\cdot\frac{1}{20}) = 6.9375\$ (97%)
In this particular case, the universal formula – excluding criticals and other special cases – is also quite simple to derive:
\$\text{New damage} = \frac{21 - \text{roll to hit} - 0.5 }{21 - \text{roll to hit}} \times \text{Old damage}\$
E.g. for a roll to hit of 16:
\$\text{New damage} = \frac{21 - \text{16} - 0.5 }{21 - \text{16}} \times \text{Old damage} = \text{0.9} \times \text{Old damage}\$
Either way, this gives you insight into...
If you have an opponent – or PC! – that is hard to hit (roll 19+), then the rule can be considered roughly equivalent to 25 % damage reduction...if the target has infinite hit points vs. the damage.
In practice, the result is harder to evaluate as HPs, damage rolls, initiative, etc.. play a factor.
In most cases I'd say the impact is still negligible. The opponents are usually in mid ranges where the impact is balanced out by the parity in damage output.
However, there are 2 noticeable exceptions that in my opinion should be considered separately.
Mook fights – If your PCs are facing a swarm of weaker creatures that have poor AB versus the AC of the party then the threat to your party can be significantly reduced.
...or...
Boss fights – If your (melee/ranged) PCs are facing an opponent with a high AC then the threat to the party can be significantly increased.
Overall, I'd say this makes the game slightly easier for the PCs. As most fights are vs. easier melee/ranged opponents (depends...depends...) and the damage is reduced more when you need higher rolls to hit the adversary, the result is that PCs will survive better.
Any time a monster is on the field for more turns than it otherwise could, hurts the players. While it isn't readily apparent during any single combat encounter, the additional combat rounds, and as a result, the additional potential damage dice against players can add up over the course of a dungeon crawl.
Monsters, over the course of a dungeon crawl, will roll more attack rolls than players. Every additional round they are on the field, increases this advantage by one set of attacks. Monsters also usually outnumber players.
This increases the percentage odds of additional damage against the party than it otherwise would have taken if it had killed the monster earlier. 1 extra chance for full damage from the monster if full damage would have killed it and half damage wouldn't.
Additional damage to the party adds up over the course of additional encounters between long rests. Meaning the party will have fewer hit points during a "Final" encounter before a long rest, or will increase the number of long rests the party has to make over the course of the adventure.
Additional combat rounds means an increase in resource expenditures. Players are more likely to cast one more spell, use one more potion, or utilize another power for every round of combat they are in.
Usually, the next set of monsters a player fights will be at full health / resources, while the players will be down hp / resources from the previous encounter.
Any time you increase raw maximum damage potential over the course of a fight (extra rounds of combat resulting in extra attacks) increases the possibility of a party wipe. Monsters (who would have otherwise died) get a raw effective / potential average DPS with each additional attack they make against a player group.
This house rule, though not as damaging as "Defender AC Wins" will subtly increase the lethality of your campaign, especially at lower levels.
