I believe your primary issue is that you have this group of users in the [Team Foundation]\Team Foundation Administrators group or possibly many users added with privileges to the Administration Console Users, which is unusual. To adjust who is part of the Team Foundation Administrators group:
- RDP to the server that hosts the application tier for your TFS installation
- Open the TFS Administration Console
- Highlight the Application Tier
- Click the Group Membership link
- Double-click
[Team Foundation]\Team Foundation Administrators
- Adjust membership of the group
To learn more about configuring administrators from the TFS Administration console, see Microsoft's documentation on the topic.