They Both Work
It's the wrong question: your child can be healthy with lots of structure or almost none.
It's About Your Situation
It's a bit too specific to your situation to answer, as it all depends on whether or not you do need the structure to live yourself within your conditions.
Likewise, you may need structure in some circumstances and not others (say, if you don't work for a given period, take a sabbathical, go on holidays...).
Do you yourself always need structure? If you feel you do, then that's probably why you'd feel like you need it for your kids. But most likely, you need that structure for a personal reason, and not because of grander reason.
Recommendations
That being said, from what you wrote I'd say you need to allow a bit more flexibility in your routine, or at least in dealing with external events. For instance, you shouldn't worry if something gets in the way, especially when going about, or you'll start worrying a lot and the children will sense that as well, and it will be less enjoyable for you as well. It's important to also show that, while structure helps, adaptability is equally important.
Also, it depends what you mean that something "gets in the way". If it's just "family lunches take longer than planned and Mary is late for her nap" or other family-related activities that "delay" or "cancel" an element of a daily routine, I'd think (but that's personal, and they're your kids, so you know best) that's overreacting. Different interactions are good, and diversity is great for young kids and infants to discover the world. So let them look at it.

Your Kids, Your Life, Your Rules
Now, about being called anal-retentive and such things no parents ever want to hear...
- If nobody ever said it and you didn't hear it indirectly, then you worrying about it probably means you already feel like you might need pushing it a bit.
- If someone ever told you, then maybe accept the criticism, all the while knowing that it's your life, your kids, and that you have to live it the way it works for you.
Oh Right, "Hard" Data
Note: to take with a grain of salt and read thoroughly before judging based on the title or abstracts only... While you could correlate parenting techniques to behaviors and successes appearing later in life, context and environment are too hard to control to imply a strong causation. That, plus any other possible bias research studies can always have and that you need to be on the lookout for (lobbying, personal biases, quality and diversity of study samples, etc...)
Also, some of these do not answer your question directly but bear a relation to the topic and how it affects other things, so I included them.