...should I be concerned?
Unless he is making these noises 'inappropriately', e.g. in the middle of a sentence when someone is speaking to him, when complete quiet is expected, if his teacher has approached you with a complaint that these noises he's making are disrupting the class, he repeatedly does it to gain attention, he does it repetitively and cannot stop, etc., then the answer is, "No."
People make noise unconsciously; it's harmless, universal, and usually serves a valid purpose. It is called nonverbal vocalization/nonlinguistic verbalization, and is often associated with emotion, although scientists don't yet quite understand the exact "whys" of them. They include sighs, laughter, moans, "hmmmms", etc. From a paper on these noises (paper concerns itself with emotional nonverbal vocalization:
The human voice is a primary tool for emotional communication. Similarly to facial expressions or body postures, nonverbal vocalizations such as laughter, crying or sighs, provide a window into the intentions and emotions of others. Nonverbal vocalizations are distinct from emotional speech regarding their underlying production and perceptual mechanisms (Pell et al., 2015; Scott, Sauter, & McGettigan, 2010), and they reflect a primitive and universal form of communication, which can be compared to the use of voice by other species...
Far less studied is the use of place holders in speech, such as um, uh, etc. But these too have an emotional aspect: to keep the attention of the audience, or discomfort with silence when speaking, etc.
So, though I couldn't find a paper specifically on nonlinguistic verbalization that wasn't related to emotion or neurological problems, it is common, and it serves a purpose. It's expressive of something. If it was detrimental, it wouldn't be as common in children (even very young babies do it) as it is.
There are times when noise should concern a parent, specifically when they seem to be "tics", uncontrolled sudden, repetitive sounds that can be hard to control. They are usually preceded by an urge to make the sound, and can be suppressed with treatment or may disappear with time.
You expressed a desire to protect your son. This noise is not something that you need to be worried about, unless someone (such as his teacher) complains about it. Relax and enjoy the individual your son is, and know that he has a life different from you, thanks to the fact that he is different from you, your parents aren't his parents, your classmates aren't his classmates, etc., and especially that you are looking out for him. But it is important that you let him be himself, and that you love and enjoy him for who he is, not for becoming someone different from that.
I used to sigh a lot, deeply and a bit noisily, during a particularly difficult (and important) academic year. It so happened I was very anxious that year. People also sigh when feeling relief, grief, and a number of other emotions. A safety stimulus signaling to [a rat] that they will not be exposed to a tail shock leads to a 20-fold increase in sighing. Poor rat.
Why Do We Say 'Hmm' When Thinking?
Automaticity in the Recognition of Nonverbal Emotional Vocalizations