BeautifulSoup get 'f slp' items from google

Question

Hi Guys I'm trying to get the citations from a number of papers out of google. This is my code

import urllib
import mechanize
from bs4 import BeautifulSoup

import csv
import os #change directory
import re #for regular expressions



br = mechanize.Browser()

br.set_handle_equiv(False)
br.set_handle_robots(False)   # ignore robots

br.addheaders = [('User-agent', 'Firefox')]             # [()]
br.open('http://google.com/')

br.select_form(name='f')   # Note: select the form named 'f' here
term = "Multinational Study of the Efficacy and Safety of Humanized Anti-HER2 Monoclonal Antibody in Women Who Have HER2-Overexpressing Metastatic Breast Cancer That Has Progressed After Chemotherapy for Metastatic Disease".replace(" ","+")
br.form['q'] = term # query
data = br.submit()

soup = BeautifulSoup(data)


cite= soup.findAll('div',{'class': 'f slp'})
ref = str(cite[1])
print ref

However I keep getting erorrs. I want the number of citations this paper has.

score 0 · Accepted Answer · edited May 23 '17 at 12:06

0

The problem is that there is no citation info on the page you are getting after the form submit, in other words there is no divs with f slp class.

You have several options to solve it:

instead of mechanize, automate it in a real browser with selenium
use google search api

BeautifulSoup get 'f slp' items from google

1 Answers1