I want to reuse the global webdriver, because its instantiation process is very slow.
And when i use driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": ua}) to define a new user-agent, it does not work, and the progress output WARNING, but everything is fine if i use single threading.
output:
ua: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2955.98 Safari/537.36
ua: Mozilla/5.0 (X11; Ubuntu; Linux i686 on x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2909.1 Safari/537.36
ua: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:47.0) Gecko/20100101 Firefox/47.0
agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:47.0) Gecko/20100101 Firefox/47.0
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: 127.0.0.1
agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:47.0) Gecko/20100101 Firefox/47.0
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: 127.0.0.1
agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:47.0) Gecko/20100101 Firefox/47.0
Here is my code, can anyone explain?
from user_agent import generate_user_agent
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import threading
options = Options()
options.headless = True
# The instantiation process is very slow, so i want to reuse the driver
driver = webdriver.Chrome(chrome_options=options, executable_path='/Applications/chromedriver')
keywords = ['play', 'the', 'piano']
def run(keyword):
ua = generate_user_agent()
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": ua})
print('ua:', ua) # This will print random user-agent
driver.get(f'https://dictionary.cambridge.org/dictionary/english/{keyword}')
agent = driver.execute_script("return navigator.userAgent")
print('agent', agent) # This always print the same user-agent
tasks = []
for i in keywords:
t = threading.Thread(target=run, args=(i,))
tasks.append(t)
t.start()
for t in tasks:
t.join()
driver.quit()