4

I haven't developed in Python in a while, and I was really excited to see pipenv enter the scene. However, I'm having some trouble using it.

I installed pipenv and then used pipenv install beautifulsoup4. My understanding is that this should have created a pipfile and a virtual env. So I started up pipenv shell. Lo and behold, my pipfile is there, with Beautiful Soup there. Next thing I tried to do was pipenv install selenium. I wrote this really short script (I'm kinda learning to do web scraping right now):

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Firefox()
profile = 'https://www.linkedin.com/in/user-profile-name'

driver.get(profile)

html = driver.page_source

soup = BeautifulSoup(html)

print(soup)

I tried running it and got this error:

Traceback (most recent call last):
  File "LiScrape.py", line 2, in <module>
    from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'

I tried running python3 in the shell and just doing import selenium to see if it would let me check the version. Again, I got the ModuleNotFoundError.

I'm so confused. What am I doing wrong with selenium that I didn't do wrong with beautiful soup??

bkula
  • 451
  • 3
  • 8
  • 21

1 Answers1

8

You just need to activate the virtual environment created by pipenv either by:

$ pipenv run python foo.py

or:

$ pipenv shell
> python foo.py

The whole process for reference:

$ pipenv --python 3.6.4 install beautifulsoup4 selenium
$ echo "import bs4 ; import selenium" > foo.py
$ pipenv run python foo.py

Or whatever version of Python you prefer.

(You should see no errors.)

This works for me.

Taylor D. Edmiston
  • 10,549
  • 5
  • 49
  • 72
  • It worked! Mostly... I got an error from selenium about 'geckodriver' executable needing to be in PATH, but your solution fixed my problem. – bkula Mar 23 '18 at 01:41
  • Awesome! This should help if you haven't resolved the PATH error already. https://stackoverflow.com/questions/40208051/selenium-using-python-geckodriver-executable-needs-to-be-in-path – Taylor D. Edmiston Mar 23 '18 at 03:38