How to find children of nodes using BeautifulSoup

Question

I want to get all the <a> tags which are children of <li>:

<div>
<li class="test">
    <a>link1</a>
    <ul> 
       <li>  
          <a>link2</a> 
       </li>
    </ul>
</li>
</div>

I know how to find element with particular class like this:

soup.find("li", { "class" : "test" })

But I don't know how to find all <a> which are children of <li class=test> but not any others.

Like I want to select:

<a>link1</a>

score 179 · Accepted Answer · edited Oct 14 '20 at 12:57

179

Try this

li = soup.find('li', {'class': 'text'})
children = li.findChildren("a" , recursive=False)
for child in children:
    print(child)

edited Oct 14 '20 at 12:57

malat

11,258
13
77
141

answered Jun 09 '11 at 02:51

cerberos

7,165
4
38
43

4

Or, to just extract the expression that describes what we want: `soup.find('li', {'class': 'text'}).findChildren()`. – Karl Knechtel Jun 09 '11 at 02:57
6

but how to get frist tag only not after wards. something like `find(li).find(a).firstChild()` – tej.tan Jun 09 '11 at 03:20
1

Thanks for the "recursive" kwarg :) – Swift May 31 '19 at 12:44
6

from bs4 site : "findChildren, findChild: These methods are left over from the Beautiful Soup 2 API. They’ve been deprecated since 2006, and should not be used at all: " – Bheid Jul 20 '21 at 08:30

score 137 · Answer 2 · edited Oct 15 '18 at 10:47

There's a super small section in the DOCs that shows how to find/find_all direct children.

https://www.crummy.com/software/BeautifulSoup/bs4/doc/#the-recursive-argument

In your case as you want link1 which is first direct child:

# for only first direct child
soup.find("li", { "class" : "test" }).find("a", recursive=False)

If you want all direct children:

# for all direct children
soup.find("li", { "class" : "test" }).findAll("a", recursive=False)

score 19 · Answer 3 · answered Jun 09 '11 at 02:48

19

Perhaps you want to do

soup.find("li", { "class" : "test" }).find('a')

answered Jun 09 '11 at 02:48

Bemmu

17,091
16
73
92

2

i think it will find ` link2 ` as well but i don't want that – tej.tan Jun 09 '11 at 02:51
2

This answers the question how to select `link1` in the HTML given in the question, but this will FAIL when the first `
` will contain no `` elements and there are other `li` elements with `test` class that contains``.

radzak

Apr 22 '18 at 16:53

This does not answer the question, but it was what I was looking for. – Pro Q Oct 13 '20 at 20:14

score 15 · Answer 4 · answered Aug 18 '14 at 02:21

15

try this:

li = soup.find("li", { "class" : "test" })
children = li.find_all("a") # returns a list of all <a> children of li

other reminders:

The find method only gets the first occurring child element. The find_all method gets all descendant elements and are stored in a list.

answered Aug 18 '14 at 02:21

kiiru

200
1
9

3

Questioner wants none of two options above. He wants all links which are only direct child. – Ahsan Roy Jul 28 '18 at 10:00

score 14 · Answer 5 · answered Apr 22 '18 at 16:46

"How to find all a which are children of <li class=test> but not any others?"

Given the HTML below (I added another <a> to show te difference between select and select_one):

<div>
  <li class="test">
    <a>link1</a>
    <ul>
      <li>
        <a>link2</a>
      </li>
    </ul>
    <a>link3</a>
  </li>
</div>

The solution is to use child combinator (>) that is placed between two CSS selectors:

>>> soup.select('li.test > a')
[<a>link1</a>, <a>link3</a>]

In case you want to find only the first child:

>>> soup.select_one('li.test > a')
<a>link1</a>

This is the one i was looking for . I was supplying it to the wrong method. Forgot > is a CSS selector. Thanks! — LFMekz, May 24 '20 at 17:29

score 8 · Answer 6 · answered Mar 04 '13 at 13:52

Yet another method - create a filter function that returns True for all desired tags:

def my_filter(tag):
    return (tag.name == 'a' and
        tag.parent.name == 'li' and
        'test' in tag.parent['class'])

Then just call find_all with the argument:

for a in soup(my_filter): # or soup.find_all(my_filter)
    print a

score 5 · Answer 7 · answered Jul 01 '21 at 07:09

Just came across this answer and checked the documentation to see that soup.findChildren is deprecated (BS 4.9). You can use soup.children instead, which only considers an element's direct children, not its descendants.

li = soup.find('li', {'class': 'text'})
for child in li.children:
    print(child)

Documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#contents-and-children

How to find children of nodes using BeautifulSoup

7 Answers7

Linked

Related