9

When trying to extract the title of a html-page I have always used the following regex:

(?<=<title.*>)([\s\S]*)(?=</title>)

Which will extract everything between the tags in a document and ignore the tags themselves. However, when trying to use this regex in Python it raises the following Exception:

Traceback (most recent call last):  
File "test.py", line 21, in <module>
    pattern = re.compile('(?<=<title.*>)([\s\S]*)(?=</title>)')
File "C:\Python31\lib\re.py", line 205, in compile
    return _compile(pattern, flags)   
File "C:\Python31\lib\re.py", line 273, in _compile
    p = sre_compile.compile(pattern, flags)   File
"C:\Python31\lib\sre_compile.py", line 495, in compile
    code = _code(p, flags)   File "C:\Python31\lib\sre_compile.py", line 480, in _code
_compile(code, p.data, flags)   File "C:\Python31\lib\sre_compile.py", line 115, in _compile
    raise error("look-behind requires fixed-width pattern")
sre_constants.error: look-behind requires fixed-width pattern

The code I am using is:

pattern = re.compile('(?<=<title.*>)([\s\S]*)(?=</title>)')
m = pattern.search(f)

if I do some minimal adjustments it works:

pattern = re.compile('(?<=<title>)([\s\S]*)(?=</title>)')
m = pattern.search(f)

This will, however, not take into account potential html titles that for some reason have attributes or similar.

Anyone know a good workaround for this issue? Any tips are appreciated.

SilentGhost
  • 287,765
  • 61
  • 300
  • 288
agnsaft
  • 1,671
  • 6
  • 28
  • 48
  • Is there some reason it has to be a zero-width assertion? Could you just use a non-capturing group? – Marcelo Cantos Apr 10 '10 at 11:47
  • 3
    Although you shouldn’t use regular expressions to process HTML. Why do you use look-arounds at all and not something like `([\s\S]*)` and take the match of the first group? – Gumbo Apr 10 '10 at 11:49

5 Answers5

12

Toss out the idea of parsing HTML with regular expressions and use an actual HTML parsing library instead. After a quick search I found this one. It's a much safer way to extract information from an HTML file.

Remember, HTML is not a regular language so regular expressions are fundamentally the wrong tool for extracting information from it.

Welbog
  • 57,620
  • 8
  • 112
  • 120
6

Here's a famous answer on parsing html with regular expressions that does a great job of saying, "don't use regex to parse html."

Community
  • 1
  • 1
Stephen Harmon
  • 845
  • 2
  • 9
  • 15
  • 4
    Yes and no. You shouldn't use regex to parse an entire DOM, or complicated nestings of tags. However, parsing a single non-nested tag, as the OP is trying to do, is a perfectly legitimate use of regex. – Cerin Mar 29 '13 at 14:57
4

The regex for extracting the content of non-nested HTML/XML tags is actually very simple:

r = re.compile('<title[^>]*>(.*?)</title>')

However, for anything more complex, you should really use a proper DOM parser like urllib or BeautifulSoup.

Cerin
  • 55,951
  • 85
  • 291
  • 487
3

What about something like:

 r = re.compile("(<title.*>)([\s\S]*)(</title>)")
 title = r.search(page).group(2)
Vojta Rylko
  • 1,402
  • 4
  • 16
  • 29
2

If you just want to get the title tag, 

html=urllib2.urlopen("http://somewhere").read()
for item in html.split("</title>"):
    if "<title>" in item:
        print item[ item.find("<title>")+7: ]
ghostdog74
  • 307,646
  • 55
  • 250
  • 337