“Underwater Combat” rules provide a framework for unfriendly environments
There are basic rules on PH p.192, under Squeezing Into a Smaller Space for dealing with creatures in spaces too small to accommodate them. These don’t go into the level of detail you are suggesting, involving restrictions on various weapons.
It sounds like you wish to create a feeling of the characters being “out of their element” in the cramped goblin warrens. The game designers accomplished this with their Underwater Combat rules. These rules can serve as a framework for any unfriendly environment.
Here is what you would do for melee weapons:
- Identify melee weapons that will work normally, and those that will have disadvantage.
- Identify creatures that are “in their element” and can ignore Rule 1.
For your goblins, the ability to use any weapon may be irrelevant, if they are using “standard issue” short swords and short bows. But you may wish to rule, for example, that any small or smaller creature has room to use any of their weapons effectively.
This is the quick and fun way to handle such environments. The party may want to adjust their tactics a bit, but (used in moderation) no player should feel frustrated about their character being made ineffective.
Useless weapons are like monster immunities
If you go beyond sorts of things in the “Underwater Combat” rules, then you may have some work to do for play balance. If you make certain weapon completely unusable in the caverns, you effectively give the denizens immunity to damage from those weapons.
The Creating a Monster: Step 9 section of the DMG (p. 277) states:
Giving a monster resistances and immunities to three or more damage types (especially bludgeoning, piercing, and slashing damage) is like giving it extra hit points.
In your case, monsters would only be immune/resistant to some weapons, but you are probably still limiting some of your PC’s damage expressions. Take your specific case into consideration (that is, how the damage expressions of the PC’s in your party would be effected) and determine if you should apply the challenge rating modifiers in the Effective Hit Points Based on Resistances and Immunities table on DMG p. 277.
Don’t spring the rules on your party
Especially if you go beyond the “Underwater Combat” style rules, make sure you give your players fair warning about whatever strictures you would impose. If a passageway is too cramped to use a PC’s favored weapon effectively, the character should realize that.
Be aware, if you make the environment too unfriendly to the party, they may simply decide not to enter.