Can any method crawl a infinite scrolling web in python without selenium?

Question

I want to crawl web page like quora, pinterest, etc. I found all examples using selenium to simulate scrolling action, yet open a new window for scrolling and crawling is slow and inefficient. Is Any method for crawling an infinite scrolling page more efficient?

score 1 · Answer 1 · edited May 23 '17 at 12:24

1

If the website uses Ajax frame to transfer data, my answer could help you. https://stackoverflow.com/a/34802775/5246180

edited May 23 '17 at 12:24

Community

1
1

answered Mar 28 '16 at 20:33

Hao Lyu

176
1
5

I've tried your method, but it can't be done in Quora. Can you show me more detail about how you did in Quora? – Leo Hsieh Apr 02 '16 at 03:11
It works on Quora but firstly you need to log in on Quora. Or it won't work. – Hao Lyu Apr 08 '16 at 20:18

Can any method crawl a infinite scrolling web in python without selenium?

1 Answers1