10

I'm using Selenium in Python to perform some web scraping in Chrome. The scraper loads a page, types a search in a searchbox, solves a basic captcha, scrapes the data, and repeats.

Problem

The chrome.exe process memory usage climbs continually, while the Python and chromedriver.exe processes have stable memory usage.

The memory leaked by Chrome is proportional to the number of pages loaded, so after around 600 pages scraped my Chrome window will run out of memory. "Google Chrome ran out of memory while trying to display this webpage."

Performance Monitor tracking process memory usage over a 10-minute period

My workaround

Workaround 1: Set a counter to load N pages before running driver.quit(), followed by driver = webdriver.Chrome() to relaunch the browser. But the memory quickly fills up again.

Workaround 2: Use Firefox with gecko webdriver instead. This has constant memory usage without a memory leak.

Version Details

OS: Windows 10 Education 1909 18363.836

Python Version: 3.8.3

Selenium Version: 3.141.0

Chrome Version: 83.0.4103.61 (Official Build) (64-bit)

ChromeDriver: 83.0.4103.39

OscarVanL
  • 626
  • 1
  • 6
  • 17
  • 6
    To the person that marked this as duplicate, this is why people don't like this site. The issue I posted here is clearly not related to the links you provided that this is a so-called duplicate of... – OscarVanL Oct 17 '20 at 12:12
  • 3
    exactly right they think they are all above. They don't even read the descriptions in the body part read the title and marked as duplicate or downvote. I have experienced a lot of this here – Darkknight Nov 07 '20 at 03:52
  • 2
    StackOverflow needs to consider this problem seriously – Darkknight Nov 07 '20 at 03:54
  • 4
    Yeah, 100% should not have been marked duplicate. I've been looking for an answer to this problem for a while. – TriHard8 Feb 25 '21 at 21:40

0 Answers0