I have seen a lot of older posts about the differences between the particles に and で. But I am surprised that neither pair in this question seems to have been asked before.
道に迷う
進路に悩む
セックスレスに悩む主婦
森で迷う
人間関係で悩む
ケーキを食べるかどうかで迷ってる
拒否と受容の間で迷う
仕事のことで悩む
「Xで」implies X is the reason for the confusion, puzzlement, or troubling. This seems an easy enough criterion. I feel roughly 60% confident about picking one over the other. But there are still cases where I am not sure about the particle.
For example, why doesn't 「人間関係に悩む」work? This may not be the best example, because I probably wouldn't make a mistake here, but I can't articulate the reason for で over に either. How about this 「セックスレスで悩む主婦」?