The georeferencer plugin is a core plugin, so you don't need to install it. You just need to enable it, which is much easier. If you need access to other (non-core) plugins, see this questions for instructions to install a plugin without using the Plugin Manager: How to install a QGIS plugin when offline?
How to enable an already installed plugin without opening the Plugin Manager:
Find your QGIS user profile file. On Windows, it should be in this folder (substitute your username):
C:\Users\username\AppData\Roaming\QGIS\QGIS3\profiles\default\QGIS
See this question for more information if you have trouble finding your user profile: Save and load user profiles in QGIS
Open the file called QGIS3.ini. It should open in a text editor, eg Notepad.
Search for "georefplugin". It should be in the [plugins] section, near the end of the file.

Make sure QGIS is closed before making any changes. (I don't actually know what happens if QGIS is open when you make changes to the active user profile file. Maybe nothing, or maybe QGIS crashes, or maybe something worse. I chose to remain ignorant.)
Change georefplugin=false to georefplugin=true. Save and close the file.
Open QGIS. The georeferencer plugin should be enabled.
Note: I actually did this on my computer, and it worked as described. I don't have any trouble opening the plugin manager, so I can't promise that this will solve your issue. Please let me know whether it works for you.
Update:
This method also worked for Cody. Their experience differed slightly from mine:
- Windows hid the directory "AppData..." when they tried to navigate to it from the file explorer. They accessed it by searching the C:\ drive.
- The "georefplugin=..." line was not present in their .ini file. They added it to the file in the position I indicated. That worked to enable the georeferencer plugin.
Thank you!
– Cody Aug 22 '19 at 20:40