If you take both the Sentinel feat and the Spear Mastery feat (from Unearthed Arcana 6 June 2016) is it possible for the enemy to get close so that they can make a melee attack against you, or do you deny it? I can't figure out how it would work in the rules.
My opinion is as soon as they start moving closer you should be able to make the attack which makes their movement speed 0 so that they cannot move closer.
Sentinel does reduce the opponent's speed to zero when you hit it with an opportunity attack. The attack given by Spear Mastery is not an opportunity attack, so Sentinel doesn't apply there.
Sentinel also doesn't mean Disengage doesn't work against you. It says:
Creatures within 5 feet of you provoke opportunity attacks from you even if they take the Disengage action before leaving your reach.
Disengage works otherwise normally, including this stipulation from the Spear Mastery ability that allows reaction attacks:
You can't use this ability if the creature used the Disengage action before moving.
Finally, the opportunity attack provided by sentinel only slows the enemy down to zero on a hit, and the probability to connect a hit is never 100%. If your opportunity attack misses, the enemy can get close as usual.
Your attack with Spear Mastery is not an opportunity attack, so it doesn't trigger Sentinel and set movement speed to 0.
If that creature moves within your spear’s reach on its next turn, you can make a melee attack against it with your spear as a reaction.
That isn't an opportunity attack - it's a reaction, but not every attack made with a reaction is an opportunity attack. To be an opportunity attack, it would need to say that it was. Sentinel only triggers on opportunity attacks, so Spear Mastery won't trigger it.
If you want to do this, the feat you need is Polearm Master. As described here, Polearm Master gives you an actual opportunity attack, triggering Sentinel and stopping creatures before they can get to you.
