Is it possible for BeautifulSoup to work in a case-insensitive manner?

Question

I am trying to extract Meta Description for fetched webpages. But here I am facing the problem of case sensitivity of BeautifulSoup.

As some of the pages have <meta name="Description and some have <meta name="description.

My problem is very much similar to that of Question on Stackoverflow

The only difference is that I can't use lxml .. I have to stick with Beautifulsoup.

For those that *can* switch from `html` to `xml` there is an answer here: https://stackoverflow.com/a/12163802/778533 — tommy.carstensen, Sep 26 '20 at 00:09

Will McCutchen · Answer 1 · 2010-04-08T19:58:18.453

18

You can give BeautifulSoup a regular expression to match attributes against. Something like

soup.findAll('meta', name=re.compile("^description$", re.I))

might do the trick. Cribbed from the BeautifulSoup docs.

edited Apr 08 '10 at 19:58

answered Apr 08 '10 at 18:26

Will McCutchen

score 17 · Answer 2 · answered Mar 07 '13 at 17:49

17

A regular expression? Now we have another problem.

Instead, you can pass in a lambda:

soup.findAll(lambda tag: tag.name.lower()=='meta',
    name=lambda x: x and x.lower()=='description')

(x and avoids an exception when the name attribute isn't defined for the tag)

answered Mar 07 '13 at 17:49

MikeyB

Using bs4 i'm getting "find_all() got multiple values for keyword argument 'name'" with that :/ – Joaolvcm Feb 20 '14 at 11:14
@Joaolvcm “You [can’t use](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#the-keyword-arguments) a keyword argument to search for HTML’s ‘name’ element, because Beautiful Soup uses the name argument to contain the name of the tag itself. Instead, you can give a value to ‘name’ in the attrs argument.” TL;DR: `soup.find_all(lambda tag: ..., {"name": lambda x: ...})`. – Alex Shpilkin Sep 21 '18 at 14:28

score 10 · Accepted Answer · answered Apr 09 '10 at 07:03

10

With minor changes it works.

soup.findAll('meta', attrs={'name':re.compile("^description$", re.I)})

answered Apr 09 '10 at 07:03

Nitin

score 6 · Answer 4 · answered May 31 '15 at 12:47

6

With bs4 use the following:

soup.find('meta', attrs={'name': lambda x: x and x.lower()=='description'})

answered May 31 '15 at 12:47

Emmanuel

score 2 · Answer 5 · edited Sep 12 '19 at 12:20

2

Better still use a css attribute = value selector with i argument for case insensitivity

soup.select('meta[name="description" i]')

edited Sep 12 '19 at 12:20

ashleedawg

answered Aug 22 '19 at 17:27

QHarr

Lucifer · Answer 6 · 2019-08-29T04:40:26.380

-5

change case of the html page source. Use functions such as string.lower(), string.upper()

edited Aug 29 '19 at 04:40

answered Aug 22 '19 at 17:16

Lucifer

why do you think so – Lucifer Aug 29 '19 at 04:40
and i am not asking question. i am answering the question – Lucifer Aug 29 '19 at 04:41

6 Answers6