This is from episode 9 of Dungeon Meshi, with the context of a person apologizing for insulting the following speaker's ill-commitment to take care of their axe:
構わん。 わしが鍛冶全般に興味がないのは事実だからな。 鉱石の見分けもつかず 昔の仲間にもよくあきれられていたものだ
It's fine. I got no interest in forging, so it's not like I'm gonna fix it. Some old buddies I had would always get on my case for not being able to tell the difference between ores. (show's translation)
I have difficulty understanding the use of 呆れる in the passive here because even if this is the "affective passive", 呆れる almost always has a negative sense by default and in the show if it wasn't passive it would be clear that the friends were dumbfounded at the speaker's lack of knowledge. Also I find it odd that it is in the past progressive because I had thought that 呆れる was basically an immediate reaction, as the English "dumbfounded at" implies as a translation.
Through this confusion I clunkily understand this as:
"My friends from a while back often used to be standing aghast (at the fact I cant distinguish ores etc.) on me"
Is my understanding correct and/or am I missing something about 呆れる, the passive or past-progressive nuances?