state of HTML after onload javascript

Question

many webpages use onload JavaScript to manipulate their DOM. Is there a way I can automate accessing the state of the HTML after these JavaScript operations?

A took like wget is not useful here because it just downloads the original source. Is there perhaps a way to use a web browser rendering engine?

Ideally I am after a solution that I can interface with from Python.

thanks!

what exactly do you want to do? I'm not sure if there's any possibility of js and python playing nicely ;D — Gordon Gustafson, Sep 17 '09 at 01:12
some screen scraping. Seems there is a Java solution: HtmlUnit — hoju, Sep 17 '09 at 04:59
seems it is possible, but difficult: http://mail.python.org/pipermail/python-list/2009-March/706800.html — hoju, Sep 17 '09 at 05:12

score 2 · Accepted Answer · answered Sep 17 '09 at 01:29

The only good way I know to do such things is to automate a browser, for example via Selenium RC. If you have no idea of how to deduce that the page has finished running the relevant javascript, then, just a real live user visiting that page, you'll just have to wait a while, grab a snapshot, wait some more, grab another, and check there was no change between them to convince yourself that it's really finished.

score 1 · Answer 2 · edited May 23 '17 at 12:11

1

Please see related info at stackoverflow:

edited May 23 '17 at 12:11

Community

1
1

answered Sep 17 '09 at 01:34

mati

5,058
3
30
47

state of HTML after onload javascript

2 Answers2

Linked