So I'm trying to use Selenium to scrape Twitter Retweets of a given Tweet. I know I can use GetRetweeters() from the Twitter API to get the last 100, but I occasionally want to go above that limit, so am trying to make a workaround.
My code is below, and the problem is as follows. Sometimes when I run it, I get all results back (12, with this example) and it works fine. More often than not, it returns maybe half of the result (six or seven), and occasionally it returns nothing. I'm running the exact same code each time.
With a bigger Tweet that has hundreds of Retweets, I've never gotten back more than eight.
Chromedriver works fine and BeautifulSoup seems to return the whole page. What's going on?
Thank you!
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome("chromedriver", options=options)
driver.get("https://twitter.com/TwitterMoments/status/1271665166216646661/retweets/without_comments")
page_source = driver.page_source
from bs4 import BeautifulSoup
soup = BeautifulSoup(page_source, "html.parser")
mydivs = soup.find_all('div', {'class':'css-901oao css-bfa6kz r-1re7ezh r-18u37iz r-1qd0xha r-a023e6 r-16dba41 r-ad9z0x r-bcqeeo r-qvutc0'})
retweeters = []
for item in mydivs:
retweeter = item.get_text()
retweeters.append(retweeter)
print(retweeters)
