This use of would instead of used to after when was very uncommon until a few decades ago. It's still not that common, but it's certainly gaining traction...

It's worth noting that there's been very little change in the relatively frequency of would and used to in contexts other than a when- clause...

Note that NGrams is case-sensitive, so there won't be many examples involving when- clauses in that second chart (just a few We used to go there when we were young). So I'm pretty confident those two charts taken together accurately reflect a usage shift specific to used to / would with when- clauses.
Because I'm older I'm not so used to would here, so I find the usage a bit distracting / irritating. But undeniably it's becoming established, so if you like it, you can reasonably ignore the fact that old farts like me don't.
Whether (and how) you convey the fact that you're talking about a habitual past act is essentially a stylistic choice. But usage is definitely changing.
Note that it's not actually necessary to use used to or would in OP's context, because it's contextually very unlikely the speaker is talking about a single visit to a restaurant. Which in any case could be ruled out simply by saying I remember when we went to fancy restaurants...