Extract form key value pairs form html page in python or parse html page

Question

I want to extract key value pairs of some form elements in a html page

for example

name="frmLogin" method="POST" onSubmit="javascript:return validateAndSubmit();" action="TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home"

while the original line is

<form name="frmLogin" method="POST" onSubmit="javascript:return validateAndSubmit();" action="TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home">

is there any method using which I can safely get the key and value pairs. I tried using splitting by spaces and then using '=' characters but string inside quotes can also have '='.

is there any different kind of split method which can also take care of quotes?

If you search for the values of the input elements of a form (as opposed to the attributes), then this answer fits: https://stackoverflow.com/a/65603777/633961 — guettli, Jan 07 '21 at 07:19

score 7 · Accepted Answer · answered Nov 17 '12 at 17:09

Use a parsing library such as lxml.html for parsing html.

The library will have a simple way for you to get what you need, probably not taking more than a few steps:

load the page using the parser
choose the form element to operate on
ask for the data you want

Example code:

>>> import lxml.html
>>> doc = lxml.html.parse('http://stackoverflow.com/questions/13432626/split-a-s
tring-in-python-taking-care-of-quotes')
>>> form = doc.xpath('//form')[0]
>>> form
<Element form at 0xbb1870>
>>> form.attrib
{'action': '/search', 'autocomplete': 'off', 'id': 'search', 'method': 'get'}

rayfranco · Answer 2 · 2012-11-17T17:07:41.373

2

You could use regular expressions like this one :

/([^=, ]+)="([^" ]+|[^," ]+)" ?"/

In python, you can do this :

#!/usr/bin/python

import re

text = 'name="frmLogin" method="POST" onSubmit="javascript:return validateAndSubmit();" action="TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home"';

ftext = re.split( r'([^=, ]+)="([^" ]+|[^," ]+)" ?', text )

print ftext;

edited Nov 17 '12 at 17:07

answered Nov 17 '12 at 16:58

rayfranco

3,492
3
25
38

Escaped double quotes could pose a problem here. – John Dvorak Nov 17 '12 at 17:05
@JanDvorak I've never seen any quotes escape in HTML, but indeed, it could be a problem... – rayfranco Nov 17 '12 at 17:13

score 1 · Answer 3 · answered Nov 17 '12 at 17:01

s = r'name="frmLogin" method="POST" onSubmit="javascript:return validateAndSubmit();" action="TG_cim_logon.asp?SID=^YcMunDFDQUoWV
32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home"'
>>> lst = s.split('" ')
>>> for item in lst:
...     print item.split('="')
... 
['name', 'frmLogin']
['method', 'POST']
['onSubmit', 'javascript:return validateAndSubmit();']
['action', 'TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Li
te_Home"']

alexvassel · Answer 4 · 2012-11-17T17:17:06.947

0

{i.split('="')[0]: i.split('="')[1] for i in str.split("\" ")}

where str is your original string

edited Nov 17 '12 at 17:17

answered Nov 17 '12 at 17:11

alexvassel

10,180
2
28
31

score 0 · Answer 5 · answered Nov 17 '12 at 17:25

dict=eval('dict(%s)'%name.replace(' ',','))
print dict
{'action': 'TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home', 'onSubmit': 'javascript:return,validateAndSubmit();', 'method': 'POST', 'name': 'frmLogin'}

This will solve your problem .

score 0 · Answer 6 · answered Jan 06 '21 at 20:29

0

You can use a library which has support for parsing HTML forms.

For example: https://mechanize.readthedocs.io/en/latest/

Stateful programmatic web browsing in Python. Browse pages programmatically with easy HTML form filling and clicking of links.

answered Jan 06 '21 at 20:29

guettli

23,964
63
293
556

Extract form key value pairs form html page in python or parse html page

6 Answers6