"Letting go" simply means to "give up craving". The Pali word is "vossagga", which means " relinquishing", "giving up", "handing over", "relaxation", "surrender", etc.
Therefore, instead of greedily trying to focus on the breathing as an object of clinging or personal security & possessiveness, you give up this greedy selfish ambition. Instead, you sit with a completely quiet & non-ambitious mind. You "die" to all ambition. You "hand over" the meditation to the "natural" relationship & functioning of the mind & body.
What will occur if you can do this is consciousness will automatically became aware of the breathing because the breathing, in relation to the quiet mind, becomes the most gross or coarse sense object, automatically.
As for "anapanasati", a Pali equivalent of the English word "of" is not found in the Pali term "anapanasati". "Anapanasati" means "mindfulness with breathing".
You did not hit the nail on the head with this question. What happened is you totally missed the boat. Because of clinging to the breathing or clinging to the bank of the river, the boat never left the shore. Instead of the current or stream of the river taking the boat to the ocean of freedom, your anchor has the boat stuck on the shore. You splash yourself with some water, as you try to row the boat frantically that is anchored to the shore, yet you believe you are a Buddhist and believe you are practising anapanasati.
Any monk, scholar, Buddhist or other that translates "anapanasati" as "mindfulness of breathing" knows close to nothing about both Pali & practise.
The Pali word "sati" or "mindfulness" means "to remember" and "keep in mind" and consciousness or knowing of breathing is not dependent on "memory".
In the suttas, knowing the breathing is called "anupassi", "pajānāti", "paṭisaṃvedī", etc, and not "sati". There is no such thing as "mindfulness of breathing".
If you are curious about why there are so few, if any, enlightened Buddhists, the ridiculous translation of "mindfulness of breathing" provides the explanation.
If you are curious why Buddhism became extinct in India, it was because of the wrong non-Buddhist yogic translation of "mindfulness of breathing", which allowed Hindu Yoga to surpass Buddhism.
If you are curious why Buddhist countries such as Sri Lanka, Thailand & Burma are so degenerate, it is because of the wrong translation "mindfulness of breathing".
If you are curious why WW1 & WW2 occurred; or why Bolshevism took over Russia; or why the US govt did 9/11, it is because wrong translation "mindfulness of breathing".
Because of the wrong translation of "mindfulness of breathing", people are simply not able to let go. Therefore, every evil of the world occurs because of this.
In summary, the role of "mindfulness" or "sati" is to remember to keep the mind in a state of "letting go". When the mind is continuously in a state of "letting go" ("vossagga"), the breathing will automatically become the sense object of consciousness.
This is why "anapanasati" means "mindfulness with breathing" rather than "mindfulness of breathing". When there is "letting go", natural automatic awareness of breathing occurs. Thus, letting go & anapanasati are inherently related. The end of the Anapanasati Sutta says:
There is the case where a monk develops mindfulness as a factor for awakening dependent on seclusion, dependent on dispassion, dependent
on cessation, resulting in relinquishment (vossagga).
"Letting go" is mentioned as the path in the Anapanasati Sutta but puthujjana believe "letting go" and "Anapanasati" to two different things.
This is the only way to stream-entry; to leave the bank of the river & reach the ocean, handing over the sailing to the current of the river and to the wind.