The game discourages using mechanics this way, but your DM decides how to handle it
The PHB says about the observable effects of damage on p. 197:
Describing the Effects of Damage
Dungeon Masters describe hit point loss in different ways. When your current hit point total is half or more of your hit point maximum, you typically show no signs of injury. When you drop below half your hit point maximum, you show signs of wear, such as cuts and bruises. An attack that reduces you to 0 hit points strikes you directly, leaving a bleeding injury or other trauma, or it simply knocks you unconscious.
The effect that damage dealt has on a creature is not only a function of the amount of damage. It is a function of the amount of damage and the number of hit points that creature has. When you deal 10 damage to a commoner, you slay that commoner on the spot. Deal the same 10 damage to a hill giant, and you will not even see a scratch.
There is the general question why the characters in the game world would not be able to infer the game mechanics over time, when they can observe their effects again and again. And there is no explicit statement in the rules that they cannot do that.
From the way the rules are written overall, addressing mostly the player when it comes to game mechanics, it however is clear that the rules are a tool for us, the players to be able to resolve the outcome of actions. They must by necessity simplify to be playable. But the idea is that they represent a world such as ours — with mundane swords, wounds and diseases, that work as they would in the real world.
The player characters generally would not know about things like hit points, they would know about being wounded or tired. The quote above shows that the severity of damage dealt is difficult to quantify for the characters — until they have dealt half a creatures hp, they cannot even see any wounds.
The DMG advises to play without thinking about the game in terms of mechanics on p. 235:
Discourage metagame thinking by giving players a gentle reminder: "What do your characters think?"
How you play this is however entirely up to you. Some tables and DMs have no problem for stating the exact numbers and sharing if it is maximal and the character acting in some way on this knowledge. You can even couch this in language like "you feel how your strike digs into the monster as if guided by a supernatural force", to provide in-game explanation to the characters.
Other tables and DMs keep all of this secret, let you roll but behind the screen record maximum damage.
I personally think that letting the character experience their special ability is at work, while keeping the monster hp secret apart from if the monster is physically wounded or maybe very close to death is a good balance, but it is really up to the individual DM and table and how they want to play.