Empty CSV when writing selenium data

Question

I'm making my first experience with Selenium and doing a few tutorials on a well-known video platform. It works quite reliable most of the time. However, I had a problem with a few pages that the CSV is created but no export of the data is made. The CSV is "touched", but it does not export the data that is displayed in a normal print.

Can anyone help me where the problem is with this script?

#_*_coding: utf-8_*_


from selenium import webdriver
import selenium
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import csv
import os

os.chdir("C:\Selenium")
PATH = "chromedriver.exe"
driver = webdriver.Chrome(PATH)

driver.get("https://twitter-trends.iamrohit.in/")

try: 
    main = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "panel-body"))
    )
 
    main = (main.text)
 
    f = open('twitter.csv', 'wb')
    print(main, file = f)
    f.close()
    
    #print(main)

except:
    driver.quit()

driver.quit()

Python Version 3.7.4, Selenium Versionm 3.141.0, Windows 10

score 1 · Accepted Answer · answered Nov 30 '21 at 15:51

1

Debugging your code you are getting the data correctly.

   main = (main.text)
   print(main)
   f = open('twitter.csv', 'wb')

So the error is when you are writing to the output file. Replacing your code for

main = (main.text)

with open('twitter.txt', 'wb', encoding='utf-8') as file1:
# Writing data to a file
    file1.writelines(main)

will work, if you check the print you have Chinese characters that will make fail the writing in the output file.

answered Nov 30 '21 at 15:51

Enrique Benito Casado

1,360
1
12
31

Thank you very much for your help. Did I understand it correctly: It will not work with Chinese characters? Is there any way to make it work? – Forelli Nov 30 '21 at 16:17
It will work if you use the option "encoding='utf-8" like i put in my code – Enrique Benito Casado Nov 30 '21 at 16:29
If you want to save your text as csv you will need to make some transformations before, you can read more here.. https://www.pythontutorial.net/python-basics/python-write-csv-file/ – Enrique Benito Casado Nov 30 '21 at 16:42
1

Now it worked... I had probably put in a bug on my first try. Perfect, thanks for the help! – Forelli Nov 30 '21 at 16:56

undetected Selenium · Answer 2 · 2021-11-30T16:21:46.467

To scrape the Twitter Trends - Worldwide table you can use DataFrame from Python Pandas and write it to a csv file using the following Locator Strategies:

Code Block:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

driver.get("https://twitter-trends.iamrohit.in/")
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//b[text()='Note:']"))))
headers = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "thead > tr > th")))]
ranks = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody//tr//descendant::th[1]")))]
topics = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[2]/a")))]
volumes = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[3]")))]
df = pd.DataFrame(data=list(zip(ranks, topics, volumes)), columns=headers)
df.to_csv(r'C:\Data_Files\output_files\twitter.csv', index=False)
driver.quit()

CSV Snapshot:

References

You can find a couple of relevant detailed discussions in:

This suggestion also works very well! Thank you! – Forelli Nov 30 '21 at 16:57 — Forelli, Nov 30 '21 at 16:57

Empty CSV when writing selenium data

2 Answers2

References

Linked