It's up to the DM.
The details of size-changing magic (including enlarge/reduce, polymorph, and Wild Shape) are not often fully explored by the rules, leaving a lot of corner-cases where the DM will have to decide how it works.
Considerations
When such an effect is mentioned within the rules, it generally either specifies that the creature's size changes as much as it can given the limits, or that the effect fails if there isn't room for it to complete. Some other spells that involve occupying a space you did not previously occupy (such as etherealness) instead shunt the creature to the nearest space that can hold it.
It seems to me that there's a tendency to treat size-changing as some sort of unstoppable force that can crush creatures to jelly or destroy even thick stone walls that get in the way. I think that's a mistake. There is no rule or metric for determining how strong a spell's size-changing power is as compared to material barriers, but it's a bit absurd to suggest that a low level spell like enlarge/reduce should be able to deal more damage through its size-changing than a purpose-made attack spell of the same level, and assumes the spell or effect has no built-in safety features that intentionally halt potentially dangerous shape-changes.
Personally, I would rule that size changing or form-changing magic fails if you try to use it while there's a Swallow-type ability still running, but forcing the creature to end its swallow effect is certainly also a valid choice. Deciding that this instantly kills either participant is probably not warranted.