2

I'm trying to extract information from Goodreads. The problem is if I go into a url like:

https://www.goodreads.com/shelf/show/programming?page=2

with Selenium chrome webdriver or with BeautifulSoup, it still shows the first page, instead of the second one.
Example with the chrome webdriver:

Example with the chrome webdriver

While on a normal browser, it displays those books instead:

enter image description here

Mohamed Oun
  • 541
  • 1
  • 7
  • 23

1 Answers1

1

Looks like that happen because you're not logged-in in your selenium session, you will have to login and save the cookies between restarts.

Take a look at this stackoverflow answers to understand how to extract cookies.

Or Duan
  • 11,648
  • 4
  • 56
  • 64