22

I am using beautiful soup and I am writing a crawler and have the following code in it:

  print soup.originalEncoding
                #self.addtoindex(page, soup)


                links=soup('a')
            for link in links:

                if('href' in dict(link.attrs)):                   
                    link['href'].replace('..', '')
                    url=urljoin(page, link['href'])
                    if url.find("'") != -1:
                        continue
                    url = url.split('?')[0]
                    url = url.split('#')[0]
                    if url[0:4] == 'http':
                        newpages.add(url)
        pages = newpages

The link['href'].replace('..', '') is supposed to fix links that come out as ../contact/orderform.aspx, ../contact/requestconsult.aspx, etc. However, it is not working. Links still have the leading ".." Is there something I am missing?

John Machin
  • 78,552
  • 11
  • 135
  • 182
sdiener
  • 243
  • 1
  • 2
  • 5

3 Answers3

58

string.replace() returns the string with the replaced values. It doesn't modify the original so do something like this:

link['href'] = link['href'].replace("..", "")
mechanical_meat
  • 155,494
  • 24
  • 217
  • 209
joel goldstick
  • 4,140
  • 6
  • 27
  • 46
15

string.replace() returns a copy of the string with characters replaced, as strings in Python are immutable. Try

s = link['href'].replace("..", '')
url=urljoin(page, s)
jan zegan
  • 1,599
  • 1
  • 10
  • 18
11

It is not an inplace replacement. You need to do:

link['href'] = link['href'].replace('..', '')

Example:

a = "abc.."
print a.replace("..","")
'abc'
 print a
'abc..'
a = a.replace("..","")
print a
'abc'
Urjit
  • 1,110
  • 8
  • 14