0

Possible Duplicate:
What's a good tool to screen-scrape with Javascript support?

I’m trying to do some screen-scraping of my bank’s website. (I know, I’m probably onto a loser, but bear with me.)

The site seems to be setting several cookies, with varying session-related values, via JavaScript, and then redirecting to the home page if it can’t find those values.

I’ve been trying to figure out a way to spot the values of those cookies by searching the HTML/JavaScript code of the pages, but the relevant code looks very obfuscated, so I’m having a hard time doing it.

Is there a Python library that simulates a web browser with JavaScript enabled? I was thinking something like mechanize that also:

  • parses the HTML page returned (e.g. with something like lxml)
  • parses any JavaScript on the HTML page
  • sets any cookies set by the JavaScript
  • amends the parsed HTML page with any DOM modifications made by the JavaScript

Basically a web browser that’s programmable in Python. Failing that, a solution in any other language.

Community
  • 1
  • 1
Paul D. Waite
  • 93,468
  • 54
  • 192
  • 264
  • 2
    Woah, so many duplicates. Sorry, I missed them on the ask page — http://stackoverflow.com/questions/125177/whats-a-good-tool-to-screen-scrape-with-javascript-support, http://stackoverflow.com/questions/1436211/state-of-html-after-onload-javascript, http://stackoverflow.com/questions/2148493/scrape-html-generated-by-javascript-with-python, http://stackoverflow.com/questions/2216826/screen-scrape-a-web-page-that-uses-javascript-and-frames, http://stackoverflow.com/questions/857515/screen-scraping-from-a-web-page-with-a-lot-of-javascript – Paul D. Waite Mar 11 '11 at 11:33
  • Anything that doesn't actually automate a browser itself is going to have a hard time against somebody with client-side knowledge determined to stop bots. Web views in mobile apps are handy because you can actually man-in-the-middle yourself and do things like pull inconvenient headers out of responses but I wouldn't recommend learning how for a hobby project. At least not objective-c/iOS if you don't already know it well. Good lord was that a pain. – Erik Reppen Feb 03 '16 at 01:39

0 Answers0