You should probably play another class
Pathfinder classes are unbalanced
One of the biggest design problems of Pathfinder is the wide power disparity between different classes. It's best summarized and conceptualized as Pathfinder class tiers. Basically, some classes (Wizard, Cleric, Druid) can do practically everything, at later levels they can do it all day long, and no problem takes them more than 24 hours to solve.
Other classes (Barbarian, Fighter, Paladin, Ranger) can do only one thing well, and fail when their niche skill doesn't work.
Classes that are considered the most balanced (usually called "Tier 3" classes), such as Inquisitor, Warpriest, or Skald, usually do one thing very well and are at least moderately useful in other situations. They will be discussed below in detail.
It's a bad idea to nerf a Fighter's ability to hit things
The class you are playing, the Fighter, only excels at hitting things with a large stick and slicing things into halves, and those halves into quarters. Literally all of your class features are dedicated to being a better melee warrior. Take this ability away, and your class is useless: you won't have any class features to work with, at all. Thus, it is a very bad idea to just make enemies unhittable.
On the other hand, hitting things is what the Fighter does really well, sometimes too well.
You can choose another class without affecting your character too much
Pathfinder features a lot of classes that are nice blends of melee and spellcasting in one form or another. They can be called “gish”, “half-casters”, and many other terms. Those are usually listed as tier 3 classes, and most of them get 6th level spells at level 20.
Such classes encompass very different concepts. Take the Magus: that's your “generic combat mage”. With the ability to cast arcane spells, many of which come from the Wizard’s list, he can also swing his one-handed weapon rather well.
The Alchemist or the Investigator, on the other hand, don’t cast any spells: instead, each day they brew special potions that only they can drink.
The Skald is your standard viking with an ability to improve melee capabilities of his friends.
Whatever concept you have, there is most likely a class in Pathfinder that does this for you. If there isn’t a class, there is an archetype.
I am not going to list all existing classes, but there is a guide that does that.
If you really don't want to use any kind of magic, there is another solution, too
If you are open to including third-party content into your game, you can try Path of War. It's an alternative rule system in which certain classes get extra abilities that are quite close to, for the lack of a better comparison, World of Warcraft classes*. They get special moves called maneuvers that are quite distinct from spells, but bring martial characters to the power level similar to Paizo tier 3 characters.
*I have only played WoW Classic and am only familiar with what Rogues and Warriors do there. I know nothing about later expansions.
Epilogue
As you could have noticed, my answer is totally focused on choosing another class instead of a Fighter. However, if it doesn't sound fun to you, or if you try it and it doesn't work well... there is no such a thing as a "wrong" kind of gaming as long as you all have fun.
So I wish that you do have fun, whatever class you play.