How to use authenticated proxy in selenium chromedriver?

Question

After searching for many hours I am starting to think this is impossible.

I need to run Chrome through selenium using different authenticated (not public) proxy's for each run.

PROXY_IP = "<some IP address>"
UID = "<the user id>"
PWD = "<the password">

options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=%s:%s@%s" % (UID,PWD,PROXY_IP))

driver = webdriver.Chrome(executable_path=".\\driver\\chromedriver.exe",
                          chrome_options=options)
driver.get("<site URL>")

Chrome will fire-up and display the error:

This webpage is not available
ERR_NO_SUPPORTED_PROXIES

If I use a public proxy requiring no authentication like this...

PROXY_IP = "<public proxy IP address>"

options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=%s" % PROXY_IP)

driver = webdriver.Chrome(executable_path=".\\driver\\chromedriver.exe",
                          chrome_options=options)
driver.get("<site URL>")

...it runs just fine and displays the site while using the proxy.

I also tried a variant with http:// in front of the user ID:

options.add_argument("--proxy-server=http://%s:%s@%s" % (UID,PWD,PROXY_IP))

The fact that I have searched far and wide and haven't found a solution leads me to believe none might exist.

I did find this but I can't make sense out of it:

selenium chromedriver authentication proxy

Not sure what browswermob-proxy is or is supposed to do or how to implement and test in Python. I hate piling up band-aid solutions unless they are absolutely necessary.

EDIT (08NOV21):

I have been away from using Selenium for many years. Because of this I now lack the context (and time, sorry) to go through the newer answers being provided and mark one as the solution to this problem. Does SO have a mechanism one could use to effectively delegate this function to someone who might be a current practitioner with expertise in this domain?

I've just submitted a bug report. https://bugs.chromium.org/p/chromedriver/issues/detail?id=1605 — jcomeau_ictx, Nov 19 '16 at 01:16

Blackster · Answer 1 · 2019-05-23T13:59:16.717

18

To use proxies with auth in python selenium you can use seleniumwire.

Fistly, install it with pip install selenium-wire

Then import webdriver from seleniumwire instead selenium

from seleniumwire import webdriver
options = {
    'proxy': {
        'http': 'http://username:password@host:port', 
        'https': 'https://username:password@host:port',
        'no_proxy': 'localhost,127.0.0.1' # excludes
    }
}
browser = webdriver.Chrome(path_to_driver, seleniumwire_options=options)

Now you can use your browser instance exact the same way as selenium: browser.get('https://api.ipify.org') and so on...

edited May 23 '19 at 13:59

answered May 23 '19 at 13:52

Blackster

183
1
5

Can i update proxy option later for same webdriver instance? – Taimur Saeed Nov 14 '19 at 13:39
@TaimurSaeed, sorry but actually I do not know. Need to test it – Blackster Nov 17 '19 at 01:14
1

@Blackster I have tested it, and you can't change the configuration once an instance is created. The work around is to create a new driver instance with different configuration – Taimur Saeed Nov 18 '19 at 07:30

score 4 · Answer 2 · answered Feb 21 '17 at 09:15

4

I have checked for most of the solutions on the web and for none of them authentication via chrome/firefox desired capabilities is working. Check this link: https://github.com/webdriverio/webdriverio/issues/324. Finally the temporary solution is to whitelist your IP address with the proxy provider.

answered Feb 21 '17 at 09:15

Anurag Narra

95
1
9

2

whitelisting it not a solution if username contains parameters – Toolkit Mar 04 '17 at 18:44

score 3 · Answer 3 · answered Jul 31 '19 at 05:00

3

This is the best solution I found and is the ONLY one that worked - all other answers on this question are outdated. It basically generates an auth extension for Chrome on the fly. Simply use the function as defined in the script as follows:

driver = proxy_chrome(host, port, username, password)
driver.get("http://www.icanhazip.com/")
driver.get("https://www.reddit.com/")
print('Terminated without issues.')

Note that this doesn't work with the --headless option. However, on Linux, you can simply use the x virtual frame buffer to simulate that. It's as easy as this in python:

import xvfbwrapper
x = xvfbwrapper.Xvfb()
x.start()

answered Jul 31 '19 at 05:00

Muppet

5,450
6
27
37

Simple example on how to use xvfbwrapper: https://gist.github.com/cgoldberg/4151516 – Spherical Cowboy Sep 21 '21 at 03:19
Also make sure to create an empty directory named 'extension' in case you use the proxy.py file linked to above, this is where the plugin file proxy_auth_plugin.zip will be stored. Moreover, host, username and password are strings, but port needs to be passed as an int. – Spherical Cowboy Sep 21 '21 at 03:22

OMERINB · Answer 4 · 2018-08-30T08:38:53.203

after trying many solutions that didn't actually work properly, i finally managed to set the authenticated proxy using the suggested extension from previous answers. what you need to do is to enter this link:

http://crxextractor.com/ and paste this url: https://www.crx4chrome.com/crx/1446/

It will let you download the extention as a .crx file without installing it. than i used this code:

proxy = {'address': 'pr.oxylabs.io:7777',
     'username': 'USERNAME',
     'password': 'PASSWORD'}

capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',
                         'httpProxy': proxy['address'],
                         'ftpProxy': proxy['address'],
                         'sslProxy': proxy['address'],
                         'noProxy': '',
                         'class': "org.openqa.selenium.Proxy",
                         'autodetect': False,
                         'socksUsername': proxy['username'],
                         'socksPassword': proxy['password']}

options = webdriver.ChromeOptions()
options.add_extension("./extension_2_0_0_0.crx")
driver = webdriver.Chrome(executable_path=CHROME_PATH, desired_capabilities=capabilities, chrome_options=options)

This solution requires manual entry of the credentials the first time it's run. — Nikolay Shindarov, Oct 03 '19 at 20:29

score 2 · Answer 5 · answered Feb 23 '19 at 08:32

I could not find any solution for chrome. We can not add extensions with headless option. I am using Heroku with chrome-buildpack. There are following options

Use xvfb instead of headless options and install extension
Use local proxy forwarder that forwards traffic to authenticated proxy; we can use Squid, mitProxy, or something like proxy-login-automator

Instead of these workaround I switched to Firefox where i was able to fill Username and Password on Proxy authentication Pop-up. Like given below. Following code is for Ruby using Capybara. You should be able to do something like this on your platform


page.driver.browser.switch_to.alert.send_keys('proxy-username' + Selenium::WebDriver::Keys::KEYS[:tab] + 'my-password')
page.driver.browser.switch_to.alert.accept

score 0 · Answer 6 · answered May 26 '15 at 18:41

0

I tryed lot of time to do the same.

Chrome is using only proxy of the OS where it is installed. You can check it by going to options-> find: proxy -> change proxy settings

So without additional addons and configuring this addons you cannot do this.

Or you can change your OS proxy settings -- this is much easier.

Also you can use phantomjs -- it have the same engine(WebKit) as chrome. using something like this:

String PROXY = proxyIP + ":" + proxyPort;
String proxyAuth= proxyUser + ":" + proxyPass;
        OpenQA.Selenium.Proxy proxy = new OpenQA.Selenium.Proxy();
                    proxy.HttpProxy = PROXY;
                    proxy.FtpProxy = PROXY;
                    proxy.SslProxy = PROXY;
                    proxy.SocksProxy = PROXY;

    var serviceJS = PhantomJSDriverService.CreateDefaultService(phantomPath);
    serviceJS.AddArguments("--proxy=" + PROXY, "--proxy-type=http", "--proxy-auth=" + proxyAuth);

answered May 26 '15 at 18:41

Andrew____Pls_Support_Ukraine

7,209
4
45
78

Chrome is NOT using OS proxy, you can easily set it by `options.AddArguments("--proxy-server=...")` – Toolkit Mar 04 '17 at 14:07
@Toolkit did you try this code and it has work? I tried to work with chrome proxy with such code and it's didn't work for me. The same is for FF browser. – Andrew____Pls_Support_Ukraine Mar 04 '17 at 15:23
yes adding proxy to ChromeDriver is trivial, `options.AddArguments("--proxy-server=http://proxy.com:111");` – Toolkit Mar 04 '17 at 15:49
3

I saw this and tried 2 years ago. More than 8 hrs tried different ways to work with proxy and no one was successful. – Andrew____Pls_Support_Ukraine Mar 04 '17 at 16:48
i am using it with C# – Toolkit Mar 04 '17 at 16:52
@Toolkit have you used socks5 proxy with authorization? I am having the same issue as the topic starter. If you have a working configuration, could you please post it somewhere? – Impworks Aug 12 '17 at 20:07
At this point it is clear that Chrome with an un-authenticating proxy is trivial. However, an authenticated proxy is not. The question is specific on using an authenticated proxy. I have been unable to locate a solution barring using an outside plugin so far. – Liquidgenius Jun 28 '18 at 17:17
> you can easily set it by options.AddArgument It is not easy as you say. Already tried @Toolkit – Volatil3 Aug 07 '18 at 08:15

score 0 · Answer 7 · answered Dec 23 '15 at 07:59

THis is a temporary solution might work in initial state: Code is in Python: Download the plugin first from chrome plugin store : Proxy-Auto-Auth_v2.0.crx

        options = webdriver.ChromeOptions()
        options.add_extension("./Proxy-Auto-Auth_v2.0.crx")) #this will provide you a window to enter user name and proxy 
        driver = webdriver.Remote(command_executor=selenium_server,desired_capabilities=options.to_capabilities())

        or 

        driver = webdriver.Chrome(chrome_options=options)

Hairy Ass · Answer 8 · 2021-11-07T14:21:00.820

I have been searching extensively. This is what it works for me



    PROXY_HOST = 'FILL IN'  # rotating proxy or host
    PROXY_PORT = 8080 # port
    PROXY_USER = 'FILL IN' # username
    PROXY_PASS = 'FILL IN' # password
    http_proxies = { 'https' : 'http://' + PROXY_USER + ':' + PROXY_PASS + '@' + PROXY_HOST + ':' + str(PROXY_PORT) }

    manifest_json = """
    {
        "version": "1.0.0",
        "manifest_version": 2,
        "name": "Chrome Proxy",
        "permissions": [
            "proxy",
            "tabs",
            "unlimitedStorage",
            "storage",
            "<all_urls>",
            "webRequest",
            "webRequestBlocking"
        ],
        "background": {
            "scripts": ["background.js"]
        },
        "minimum_chrome_version":"22.0.0"
    }
    """

    background_js = """
    var config = {
            mode: "fixed_servers",
            rules: {
            singleProxy: {
                scheme: "http",
                host: "%s",
                port: parseInt(%s)
            },
            bypassList: ["localhost"]
            }
        };

    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

    function callbackFn(details) {
        return {
            authCredentials: {
                username: "%s",
                password: "%s"
            }
        };
    }

    chrome.webRequest.onAuthRequired.addListener(
                callbackFn,
                {urls: ["<all_urls>"]},
                ['blocking']
    );
    """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)




    def get_chromedriver(self, use_proxy=False):

        driver_path = 'FILL IN'
        path = os.path.dirname(os.path.abspath(__file__))
        chrome_options = webdriver.ChromeOptions()
        if use_proxy:
            pluginfile = 'proxy_auth_plugin.zip'

            with zipfile.ZipFile(pluginfile, 'w') as zp:
                zp.writestr("manifest.json", libshared.manifest_json)
                zp.writestr("background.js", libshared.background_js)
            chrome_options.add_extension(pluginfile)

        driver = webdriver.Chrome(
            executable_path=driver_path,
            chrome_options=chrome_options)
        return driver

V-cash · Answer 9 · 2021-12-02T20:53:40.027

1º If you always use the same IP you can speak with your proxy provider to access proxy with your IP = no user or password needed. In my experience that's easier.

If that's not possible other solution is: pip install undetectable chromedriver and seleniumwire if you don't have them

selenium-wire 4.5.6 selenium 4.1.0 undetected-chromedriver 2.2.1

from seleniumwire.undetected_chromedriver.v2 import Chrome

options = {
    'proxy': {
        'http': 'http://user:pass@ip:port',
        'https': 'https://user:pass@ip:port',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = Chrome(seleniumwire_options=options)

or you can also try another solution:

import undetected_chromedriver.v2 as uc

IP = XXXX
SOCKS5_PORT = XXXX
HTTPS_PORT = XXXX

options = uc.ChromeOptions()
options.add_argument(f'--proxy-server=socks5://{IP}:{SOCKS5_PORT}')
#you can substitue the last line for this one:
options.add_argument(f'--proxy-server=https://{IP}:{HTTPS_PORT}')

driver = uc.Chrome(executable_path=EXECUTABLE_PATH, options=options)

Thanks for the reply. Please read my edit with regards to no-longer being in a position to evaluate answers. Still, I hope your answer and those of others might help those with similar issues. It might very well help me in the future if I ever come back to this domain. — martin's, Dec 01 '21 at 02:40

How to use authenticated proxy in selenium chromedriver?

9 Answers9

Linked