How can we use Selenium Webdriver in colab.research.google.com?

Question

I want to use Selenium Webdriver of Chrome in colab.research.google.com for fast processing. I was able to install Selenium using !pip install selenium but the webdriver of chrome needs a path to webdriverChrome.exe. How am I suppose to use it?

P.S.- colab.research.google.com is an online platform which provides GPU for fast computational problems related to deep learning. Please refrain from solutions such as webdriver.Chrome(path).

I think I mentioned "colab.research.google.com". I know how webdriver works on a local machine. But as colab research google is an online platform which provides GPU for fast machine learning processing problems, I want to use webdrive on this above mentioned online platform. — john mich, Jun 26 '18 at 16:52
A same problem is in this link: https://stackoverflow.com/questions/54327654/how-can-i-insert-path-environmental-variable-for-geckodriver-in-goggle-colab — Hamed Baziyad, Jan 23 '19 at 13:42
@Dimanjan Hey, I have stopped trying this. Use-case was scrapped and so did not explored further. — john mich, Jul 10 '19 at 09:24

score 93 · Accepted Answer · edited Jul 07 '19 at 11:23

93

You can do it by installing the chromium webdriver and adjusting some options such that it does not crash in google colab:

!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.get("https://www.webite-url.com")

edited Jul 07 '19 at 11:23

mnimmny

3
3

answered Jan 07 '19 at 16:07

Thomas

1,496
7
21

4

Are you have such suggestion for gecko driver? – Hamed Baziyad Jan 23 '19 at 08:41
Hi, could you please the question in this link: https://stackoverflow.com/questions/54327654/how-can-i-insert-path-environmental-variable-for-geckodriver-in-goggle-colab – Hamed Baziyad Jan 23 '19 at 13:41
Thanks for sharing, I will try this and let you know – john mich Jan 31 '19 at 18:33
cp: cannot stat '/usr/lib/chromium-browser/chromedriver': No such file or directory – vesszabo Jun 27 '19 at 18:38
1

This solution works as of 6/4/21. Thank you! – user185160 Jun 04 '21 at 18:22

score 25 · Answer 2 · edited Sep 07 '21 at 18:30

25

this one worked in colab

!pip install selenium
!apt-get update 
!apt install chromium-chromedriver

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)

edited Sep 07 '21 at 18:30

Red Headphone

27
6

answered Apr 11 '20 at 13:59

Shaina Raza

1,095
12
10

korakot · Answer 3 · 2022-04-28T18:06:56.813

20

I made my own library to make it easy.

!pip install kora -q
from kora.selenium import wd
wd.get("https://www.website.com")

PS: I forget how I searched and experimented until it worked. But I first wrote and shared it in this gist in Dec 2018.

edited Apr 28 '22 at 18:06

answered Jul 16 '20 at 07:12

korakot

32,074
13
108
128

2

This is amazing. Thanks a lot for this. – john mich Jul 16 '20 at 17:27
2

Doesn't work on google colab: gives the same error: WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: -6 – Ahmad M. Sep 24 '20 at 15:49
1

@AhmadM. It should be working now. URL was not correct at first. – korakot Sep 24 '20 at 16:09
Indeed, this is a brilliant library ! – Ahmad M. Sep 25 '20 at 14:09
@korakot is there a support for the tor browser in your library? – john mich Apr 09 '21 at 10:35
@johnmich No. Only chrome-selenium is supported. – korakot Apr 09 '21 at 12:54
@korakot `WebDriverWait` is supported there? – αԋɱҽԃ αмєяιcαη Jul 12 '21 at 05:34
It worked nicely a few months ago, but now it doesn't with selenium 4.1.0 : `wd.get("https://www.twitter.com")` --> `JavaScript is not available.`. Is there a way to enable javascript in the webdriver ? – stallingOne Nov 26 '21 at 14:57

score 7 · Answer 4 · answered Oct 06 '21 at 11:38

Don't have enough repu to comment. :(

However @Thomas answer still works in 06.10.2021, but with just one simple change since right of the bat you'll get DeprecationWarning: use options instead of chrome_options

Working code below:

!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',options=options)
wd.get("https://stackoverflow.com/questions/51046454/how-can-we-use-selenium-webdriver-in-colab-research-google-com")
wd.title

If you don't have the reputation to comment, you should instead edit q/a and write answers until you have the reputation. — CATboardBETA, Oct 06 '21 at 13:19
I mean write actual answers to *other* questions, that aren't practically comments. — CATboardBETA, Oct 07 '21 at 12:09

score 2 · Answer 5 · answered Dec 02 '21 at 12:26

to use selenium in GOOGLE COLAB do the next steps in the colab notebook

!pip install kora -q

HOW TO USE IT INSIDE COLAB :

from kora.selenium import wd
wd.get("enter any website here")

YOU CAN ALSO USE IT WITH Beautiful Soup

import bs4 as soup
wd.get("enter any website here")
html = soup.BeautifulSoup(wd.page_source)

score -3 · Answer 6 · answered Jun 26 '18 at 18:31

You can can rid of using .exe file by using WebDriverManager so instead of this

System.setProperty("webdriver.gecko.driver", "driverpath/.exe");
WebDriver driver = new FirefoxDriver();

you will be writing this

WebDriverManager.firefoxdriver().setup();
WebDriver driver = new FirefoxDriver();

All you need is add the dependecy to the POM file(Im assuming you using maven or some build tool) Please see my full answer about how to use this in this link Using WebdriverManager

You run `python` on Colab, not `javascript`. Please give a python answer. — korakot, Jun 27 '18 at 09:11

How can we use Selenium Webdriver in colab.research.google.com?

6 Answers6

Linked