Parsing HTML page containing JS in Java

Question

I am trying to parse a web page, which contains some JS. Till now I am using Jsoup to parse html in Java, which is working as expected. But I am unable to parse the JavaScript. Below is the snippet of the HTML page-

<script type="text/javascript"> 
var element = document.createElement("input"); 
element.setAttribute("type", "hidden");
element.setAttribute("value", "");
element.setAttribute("name", "AzPwXPs");
element.setAttribute("id", "AzPwXPs");
var foo = document.getElementById("dnipb"); 
foo.appendChild(element);
var element1 = document.createElement("input"); 
element1.setAttribute("type", "hidden");
element1.setAttribute("value", "6D6AB8AECC9B28235F1DE39D879537E1");
element1.setAttribute("name", "ZLZWNK");
element1.setAttribute("id", "ZLZWNK");
foo.appendChild(element1);
</script>

I want to read both the values with their name/id. So that after parsing I can get following results-

AzPwXPs=
ZLZWNK=6D6AB8AECC9B28235F1DE39D879537E1

How to parse in this situation?

@nhahtdh: Ya, I know that. That is why I am stuck in between... :( But there must some other way around — Ravi Joshi, May 01 '13 at 10:49
Run it through a JS parser? Or get a JS engine? (I actually also have the same problem on a side project, but I never got my hand around it...) — nhahtdh, May 01 '13 at 10:52

score 6 · Answer 1 · answered Mar 02 '17 at 08:25

I have stumbled upon this question few times when searching for the solution to parse pages with JavaScript but the solution provided is not perfect. I have found pure Java solution to the problem by using JBrowserDriver and JSoup to parse JavaScript manipulated page.

Simple example:

    // JBrowserDriver part
    JBrowserDriver driver = new JBrowserDriver(Settings
            .builder().
            timezone(Timezone.EUROPE_ATHENS).build());
    driver.get(FETCH_URL);
    String loadedPage = driver.getPageSource();

    // JSoup parsing part
    Document document = Jsoup.parse(loadedPage);
    Elements elements = document.select("#nav-console span.data");

    log.info("Found element count: {}", elements.size());

    driver.quit();

Hugo Pedrosa · Answer 2 · 2018-06-28T09:16:30.693

2

I already had the same situation to find url's in css files.

Put the javascript in a string and a apply Regular expressions

Pattern p = Pattern.compile("url\\(\\s*(['" + '"' + "]?+)(.*?)\\1\\s*\\)"); //expression
Matcher m = p.matcher(content);
while (m.find()) {
String urlFound = m.group(); 
}

Regards, Hugo Pedrosa

edited Jun 28 '18 at 09:16

answered May 02 '13 at 16:07

Hugo Pedrosa

336
2
12

Although I got the logic but how this can be modified to serve my purpose? – Ravi Joshi May 02 '13 at 17:07

score 1 · Answer 3 · answered May 02 '13 at 15:54

1

You can try using query library. Its much more easier with it.

answered May 02 '13 at 15:54

Java Israel

11
1

1

Can you please suggest any one of them? Is it available in Java? – Ravi Joshi May 02 '13 at 17:06

score 1 · Accepted Answer · edited May 23 '17 at 12:00

1

Selenium's Webdriver is fantastic: http://docs.seleniumhq.org/docs/03_webdriver.jsp

See this answer for an example of what you are trying to do: Using Selenium Web Driver to retrieve value of a HTML input

edited May 23 '17 at 12:00

Community

1
1

answered May 02 '13 at 16:11

Tom Carchrae

6,194
2
34
35

What you mean by jQuery is being used? Can you explain me a little bit more please? – Ravi Joshi May 02 '13 at 17:05
the javascript `return $('#AzPwXPs')[0]` uses a jquery selector ( $('#AzPwXPs')) to find the element. – Tom Carchrae May 02 '13 at 17:27
So, finally in my java parser project there will be `Selenium's Webdriver` with `jquery`. Is it? – Ravi Joshi May 02 '13 at 18:24
no, you don't need to use jquery - i'm just going to link to an answer that shows you a better example – Tom Carchrae May 02 '13 at 20:19

score 1 · Answer 5 · edited May 23 '17 at 12:00

1

Once you've got the text content of the <script> element from JSoup, you can parse the JS using the Caja JS parser and then walk the parse tree to find what you're looking for.

edited May 23 '17 at 12:00

Community

1
1

answered May 02 '13 at 16:15

Mike Samuel

114,030
30
209
240

How to get ` – Ravi Joshi May 02 '13 at 17:03
1

@Ravi, If you're doing something like `Document doc = Jsoup.parse(...)`, then `doc.getElementsByTag("script").first()` should get you the first script in the page. – Mike Samuel May 02 '13 at 17:29

Parsing HTML page containing JS in Java

5 Answers5

Linked