Trying to get table rows with Scrapy xpath

Question

I have some html that looks like the screenshot. I want to get the table rows. I have:

for table_row in response.selector.xpath("//*[@id = 'ctl00_ContentPlaceHolder1_CaseDetailParties1_gvParties']"):
    print table_row

In the command line I tried:

>>> table_row
Out[5]: <Selector xpath="//*[@id = 'ctl00_ContentPlaceHolder1_CaseDetailParties1_gvParties']" data=u'<table class="ParamText" cellspacing="0"'>
>>> table_row.xpath('/tbody')
Out[6]: []
>>> table_row.xpath('//tbody')
Out[7]: []

Why am I unable to select the tbody?

score 7 · Accepted Answer · edited May 23 '17 at 12:31

7

tbody is generated by the browser, you don't get it with Scrapy downloader. Just get straight to the tr elements:

table_row.xpath('.//tr')

edited May 23 '17 at 12:31

Community

1
1

answered Oct 19 '16 at 15:49

alecxe

441,113
110
1,021
1,148

1

Wow, I thought I was going crazy. Thank you! – user1592380 Oct 19 '16 at 15:56
If I can, I have a follow up: There are multiple nested tables inside the tr elements in this table.How can I get just the children tr's in other words, table_row.xpath('//tbody/tr') if tbody existed – user1592380 Oct 19 '16 at 16:06
@user61629 ah, you should then filter top level `tr`s, if I understand you correctly, e.g. `table_row.xpath('./tr')`.. – alecxe Oct 19 '16 at 16:09
Yes thats it. Thank you very much! – user1592380 Oct 19 '16 at 16:21
man, you saved my day ! Thank you – A. Maitre Dec 02 '21 at 14:32

Trying to get table rows with Scrapy xpath

1 Answers1