-1

Hi I have been trying to extract specific text from a class element and click the button named "Next >" and continue extracting unless there is no similar button found in the page.

Here is the current implementation:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
import pandas as pd
import re

def myClick(by, desc):
    wait = WebDriverWait(browser, 10)
    by = by.upper()
    if by == 'XPATH':
        wait.until(EC.element_to_be_clickable((By.XPATH, desc))).click()
    if by == 'ID':
        wait.until(EC.element_to_be_clickable((By.ID, desc))).click()
    if by == 'LINK_TEXT':
        wait.until(EC.element_to_be_clickable((By.LINK_TEXT, desc))).click()


browser = webdriver.Chrome('chromedriver.exe')
browser.get("https://www.google.com/localservices/prolist?g2lbs=AL1YbfWFoliLAcUoULk2nolUcxDjNfIqkgyv0l7PEt80QYPYYu10xpBoxGeRB9Fkbt9oFChPgdmgoWouRDXzeJQlzehVQxwZIJs_OU_qbdnZyou62ovD17M%3D&hl=en-DK&gl=dk&ssta=1&oq=advokat&src=2&origin=https%3A%2F%2Fwww.google.com&sa=X&slp=MgBSBggCGgAgAEAB&scp=Cg1nY2lkOmxhd19maXJtEkgSEgn7X5TutidLRhHNkuDD0EOHUhoSCftflO62J0tGEc2S4MPQQ4dSIghEZW5tYXJrICoUDUmecSAVwZuuBB24cJYiJRDiEwkaB2Fkdm9rYXQqCExhdyBmaXJt&q=advokat%20near%20Denmark%20&ved=0CAAQ28AHahcKEwjA9Zju0vz3AhUAAAAAHQAAAAAQNg")
# scrape some data on a page and extract table

next_button = browser.find_element_by_xpath("//span[contains(text(),'Next >')]") #this element is visible
# next_button = WebDriverWait(browser, 2).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="yDmH0d"]/c-wiz[3]/div/div[3]/div/div/div[1]/div[3]/div[3]/c-wiz/div/div/div[2]/div/div/button/span')))
# browser.execute_script("arguments[0].click();", WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="yDmH0d"]/c-wiz/div/div[3]/div/div/div[1]/div[3]/div[3]/c-wiz/div/div/div[2]/div/div/button'))))

# next_button = myClick('xpath', '/html/body/c-wiz[3]/div/div[3]/div/div/div[1]/div[3]/div[3]/c-wiz/div/div/div[2]/div/div/button/span')
company_names = []

while next_button.is_displayed():
    src = browser.page_source
    soup = BeautifulSoup(src,'html.parser')
    next_button = browser.find_element_by_xpath("//span[contains(text(),'Next >')]")
    print("NEXT \n", next_button)
    innerContent = soup.find_all(class_=re.compile("rgnuSb xYjf2e"))
    print(innerContent)
    for ul in innerContent:
        print("UL-----", ul.text)
        company_names.append(ul.text)
        print("*********************")
    
    next_button.click()

browser.close()

It gives me the following error:

raise exception_class(message, screen, stacktrace) selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element ... is not clickable at point (19, 9). Other element would receive the click: ...

However, if you run the script you will observe that it cannot paginate after second page and also cannot extract any text element from the class after the first page. I tried some way to capture the click based on some Stackoverflow Q/A : In python selenium, how does one find the visibility of an element?, Selenium - wait until element is present, visible and interactable

Can anyone please tell me what I am doing wrong?

Shihab Ullah
  • 61
  • 5
  • That is in the eye of the beholder, not to be prevented and yes a little explanation would always be helpful. but don't be sad about it, it surely happened to everyone here before. here are a few clues from a discussion about this question: https://meta.stackoverflow.com/questions/252677/when-is-it-justifiable-to-downvote-a-question – HedgeHog May 27 '22 at 07:33
  • @HedgeHog, can you help me with the answer or is it possible to tag someone? Kindly upvote the question if it make sense to you. Thanks in advance – Shihab Ullah May 27 '22 at 08:03
  • 1
    Would have done it already if I could think of a solution for it, sometimes it just takes some time until someone comes across your question and has a suitable solution for it – HedgeHog May 27 '22 at 08:49

0 Answers0