0

I want to reuse the global webdriver, because its instantiation process is very slow. And when i use driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": ua}) to define a new user-agent, it does not work, and the progress output WARNING, but everything is fine if i use single threading.

output:

ua: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2955.98 Safari/537.36
ua: Mozilla/5.0 (X11; Ubuntu; Linux i686 on x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2909.1 Safari/537.36
ua: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:47.0) Gecko/20100101 Firefox/47.0
agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:47.0) Gecko/20100101 Firefox/47.0
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: 127.0.0.1
agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:47.0) Gecko/20100101 Firefox/47.0
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: 127.0.0.1
agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:47.0) Gecko/20100101 Firefox/47.0

Here is my code, can anyone explain?

from user_agent import generate_user_agent
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import threading

options = Options()
options.headless = True

# The instantiation process is very slow, so i want to reuse the driver
driver = webdriver.Chrome(chrome_options=options, executable_path='/Applications/chromedriver')   

keywords = ['play', 'the', 'piano']


def run(keyword):
    ua = generate_user_agent()
    driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": ua})
    print('ua:', ua)  # This will print random user-agent
    driver.get(f'https://dictionary.cambridge.org/dictionary/english/{keyword}')
    agent = driver.execute_script("return navigator.userAgent")
    print('agent', agent)  # This always print the same user-agent
    

tasks = []    
for i in keywords:
    t = threading.Thread(target=run, args=(i,))
    tasks.append(t)
    t.start()

for t in tasks:
    t.join()
    
driver.quit()
fitz
  • 500
  • 2
  • 9
  • This answer helped me:https://stackoverflow.com/questions/30808606/can-selenium-use-multi-threading-in-one-browser – fitz Sep 15 '21 at 01:01

0 Answers0