0

I am trying to scrape a js website with selenium. When beautiful soup reads what selenium retrieved I get an html page that says: "Cookies must be enabled in order to view this page." If anyone could help me past this stumbling block I would appreciate it. Here is my code:

# import libraries and specify URL
import lxml as lxml
import pandas as pd
from bs4 import BeautifulSoup
import html5lib
from selenium import webdriver
import urllib.request
import csv

url = "https://racing.hkjc.com/racing/information/English/Racing/LocalResults.aspx?RaceDate=2020/06/09"

#new chrome session
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
driver = webdriver.Chrome(executable_path= '/Users/susanwhite/PycharmProjects/Horse 
Racing/chromedriver', chrome_options=chrome_options)



# Wait for the page to fully load
driver.implicitly_wait(time_to_wait=10)

# Load the web page
driver.get(url)
cookies = driver.get_cookies()




# Parse HTML code and grab tables with Beautiful Soup
soup = BeautifulSoup(driver.page_source, 'html5lib')
print(soup)
Spiker
  • 1

1 Answers1

0

Try removing this line: chrome_options.add_argument("--incognito"). There's no need for it, as Selenium naturally doesn't save cookies or any other information from websites.

hymn0
  • 1
  • 1
  • Thank you for getting back to me. The incognito mode was just me trying different options to see if I could get something to work. I tried both with and without and got the same results. – Spiker Mar 27 '22 at 15:57
  • No problem. I gave another answer to a problem that looks like yours. If you wanna check it: https://stackoverflow.com/questions/71634382/how-to-accept-span-button-using-selenium/71635019#71635019 – hymn0 Mar 27 '22 at 19:01
  • I used some of the options that you had in the above answer, and I got it to work. Once. After that It gives me a "SessionNotCreatedException: Message: session not created" error – Spiker Mar 28 '22 at 01:10
  • Try using a newer version of Chrome on the user agent, or checking this answer: https://stackoverflow.com/questions/60296873/sessionnotcreatedexception-message-session-not-created-this-version-of-chrome – hymn0 Mar 28 '22 at 02:28