3

It's not really scraping, I'm just trying to find the URLs in a web page where the class has a specific value. For example:

<a class="myClass" href="/url/7df028f508c4685ddf65987a0bd6f22e">

I want to get the href value. Any ideas on how to do this? Maybe regex? Could you post some example code? I'm guessing html scraping libs, such as BeautifulSoup, are a bit of overkill just for this...

Huge thanks!

Charles Stewart
  • 11,453
  • 4
  • 45
  • 84
pns
  • 413
  • 1
  • 7
  • 18

7 Answers7

16

Regex is usally a bad idea, try using BeautifulSoup

Quick example:

html = #get html
soup = BeautifulSoup(html)
links = soup.findAll('a', attrs={'class': 'myclass'})
for link in links:
    #process link
Yacoby
  • 52,948
  • 13
  • 109
  • 118
9

Aargh, not regex for parsing HTML!

Luckily in Python we have BeautifulSoup or lxml to do that job for us.

Community
  • 1
  • 1
Daniel Roseman
  • 567,968
  • 59
  • 825
  • 842
2

Regex would be a bad choice. HTML is not a regular language. How about Beautiful Soup?

Mark Byers
  • 767,688
  • 176
  • 1,542
  • 1,434
1

Regex should not be used to parse HTML. See the first answer to this question for an explanation :)

+1 for BeautifulSoup.

Community
  • 1
  • 1
John Keyes
  • 5,329
  • 1
  • 27
  • 45
1

If your task is just this simple, just use string manipulation (without even regex)

f=open("htmlfile")
for line in f:
    if "<a class" in line and "myClass" in line and "href" in line:
        s = line [ line.index("href") + len('href="') : ]
        print s[:s.index('">')]
f.close()

HTML parsers is not a must for such cases.

ghostdog74
  • 307,646
  • 55
  • 250
  • 337
0

read Parsing Html The Cthulhu Way https://blog.codinghorror.com/parsing-html-the-cthulhu-way/

Cœur
  • 34,719
  • 24
  • 185
  • 251
George Godik
  • 1,716
  • 1
  • 14
  • 19
0

The thing is I know the structure of the HTML page, and I just want to find that specific kind of links (where class="myclass"). BeautifulSoup anyway?

pns
  • 413
  • 1
  • 7
  • 18