The rules are intentionally silent
What do the rules say should happen in this situation?
The rules do not describe this situation. Instead, the rules imply the DM will decide:
"... the rules can't cover everything. When you encounter something that the rules don't cover or if you're unsure how to interpret a rule, the DM decides how to proceed"
Tasha's Cauldron of Everything page 4, "The DM Adjudicates the Rules"
It's the DM's job to make decisions and adjudications during the game. The lead game developer supports this approach, using a very similar example:
What happens when someone swallows a druid in a Tiny form? <...> The rules are intentionally silent on these corner cases, leaving adjudication to DMs.
This is a part of the larger principle "rulings over rules" from earlier editions, which 5e tries to resurrect. See Where have the designers of 5e said that they favored rulings over rules in this edition?
So what are the options?
As a DM, you have a few options. Choose one depending on the player's intent, your own play style, common sense, and table conventions:
- Shapechanging into a gargantuan creature kills the swallower instantly. Can be probably abused, so use it with care. Kind of gross. Can also be used against PCs as well (ask the players, if they are okay with that).
- The same, but the caster also takes damage. A trade off for an insta-kill. Still gross.
- A caster can't Shapechange if there is no space. The magic doesn't work like that. A polite way of saying "no". Fair but harms creativity.
- The middle way — the swallower throws up, probably suffer damage in the process. Probably the optimal one, kind of boring.
Choose one and stick to it. Consistency is the key — players doing a same thing should get the same (expected) results. More info: What is Player Agency and what is it good for?
We play games to have fun, so players' (and the DM's) fun is the first priority. This makes the "Rule of Cool" completely legit. See the D&D Adventurers League: DMS for details:
Run a fun game! As the DM, you help guide the players through the adventure, and facilitate the story coming alive. Make sure you follow the golden rule: Make decisions and adjudications during the game that enhance the fun when possible.
Also on-topic:
Inspirations: