I have a set of web scrapers currently running on my personal PC using the Chrome web driver. I'd like to move them into a virtual environment and run Chrome in headless mode, but I'm running into weird HTML/JS errors.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
options = Options()
# options.headless = True
driver = webdriver.Chrome(options=options)
driver.get('https://facdissem.census.gov/SearchA133.aspx')
driver.find_element(By.CSS_SELECTOR, '#ui-id-1').click()
print('clicked!')
The code above works fine as long as line 6 is commented out. However, if I uncomment it, I get the following logs:
[0304/163706.130:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/jquery-1.11.2.min.js (1)
[0304/163706.616:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/jquery-ui-1-11-4.min.js (1)
[0304/163706.740:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/ims_jscript.js (1)
[0304/163706.740:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/gov.census.FAC.Utilities.js (1)
[0304/163706.741:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/gov.census.FAC.Controls.js (1)
[0304/163706.742:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/gov.census.FAC.ToolTips.js (1)
[0304/163706.744:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/shortcut.js (1)
[0304/163707.227:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/jquery.cookie.js (1)
[0304/163707.229:INFO:CONSOLE(57)] "Uncaught ReferenceError: shortcut is not defined", source: https://facdissem.census.gov/SearchA133.aspx (57)
[0304/163707.350:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/WebResource.axd?d=x2tleshr6wUlOuIvJRSEnb9PfqpNNy9816HQxyOe9H-OWnyj667vrgEpSImZx0rvxkClm_X1c6C7XJfimZ8v2HlyOUdOfqLvGVB46bH-zts1&t=637729441680000000 (1)
[0304/163707.350:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/gov.census.FAC.ToolTips.js (1)
[0304/163707.372:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/Scripts/gov.census.FAC.CheckableCollection.js (1)
[0304/163707.373:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/ScriptResource.axd?d=R-Nnn6PCCMGOOZ_ro2SJZ0c0LtSShWd7SjIg-Q68U-2VWBRIcIQQMBRl_gVEjQg8qKmAYNIEuPxAmwcGL7XR5KQD_irMubY442_dSSAe9oOYjlpXCFofh8-QlR6PjkHOdiHxSFMVrZHzz6cYMACqmoVsno3DJm47tLHdUWWBLSg1&t=ffffffffd202a464 (1)
[0304/163707.373:INFO:CONSOLE(1)] "Uncaught SyntaxError: Unexpected token '<'", source: https://facdissem.census.gov/ScriptResource.axd?d=0XAAYVSO34nE2qDEReSoBKm7lX7dgelJNq75wq6uwx7px3Qiof4y0Hi3x00KnwwWoS4RWuTACJRUOmX3lGDSzILa9kDDJAZBxJP2f3vxVfQsd9ErCR9yJyvM35G810dR-5yItqenCTfNW90zWj_Dw0uC4wXoqJqnftEnSkybCtpudVneyoTW4G_Bpc7n41OC0&t=ffffffffd202a464 (1)
[0304/163707.375:INFO:CONSOLE(383)] "Uncaught ReferenceError: Sys is not defined", source: https://facdissem.census.gov/SearchA133.aspx (383)
[0304/163707.376:INFO:CONSOLE(445)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (445)
[0304/163707.377:INFO:CONSOLE(727)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (727)
[0304/163707.432:INFO:CONSOLE(787)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (787)
[0304/163707.432:INFO:CONSOLE(855)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (855)
[0304/163707.460:INFO:CONSOLE(974)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (974)
[0304/163707.469:INFO:CONSOLE(1284)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (1284)
[0304/163707.473:INFO:CONSOLE(2008)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (2008)
[0304/163707.476:INFO:CONSOLE(2582)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (2582)
[0304/163707.479:INFO:CONSOLE(3159)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (3159)
[0304/163707.516:INFO:CONSOLE(3239)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (3239)
[0304/163707.532:INFO:CONSOLE(3935)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (3935)
[0304/163707.549:INFO:CONSOLE(4006)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (4006)
[0304/163707.551:INFO:CONSOLE(4145)] "Uncaught ReferenceError: $ is not defined", source: https://facdissem.census.gov/SearchA133.aspx (4145)
...
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#ui-id-1"}
...
The first thing I thought was that something was wrong with the HTML, so I dumped the pages I received in headless and normal Chrome modes, but it turned out that their content is identical.
I have tried randomly adding many other options as suggested in the answers to other questions, such as:
# options.add_argument('--window-size=1920,1080')
# options.add_argument('--start-maximized')
# options.add_argument('--no-sandbox')
# options.add_argument('--disable-gpu')
# options.add_argument('--ignore-certificate-errors')
# options.add_argument('--disable-extensions')
# options.add_argument('--disable-dev-shm-usage')
... but I still get the same output.
Environment
Win 10
Python 3.10.1
Selenium 4.1.2
Chrome 99.0.4844.51
ChromeDriver 99.0.4844.51
Is there anything I can do about this or is the only way I have is to use remote web driver with Chrome running in a Docker container?
