Python: Split a string, respect and preserve quotes

Question

Using python, I want to split the following string:

a=foo, b=bar, c="foo, bar", d=false, e="false"

This should result in the following list:

['a=foo', 'b=bar', 'c="foo, bar"', 'd=false', 'e="false'"']

When using shlex in posix-mode and splitting with ", ", the argument for cgets treated correctly. However, it removes the quotes. I need them because false is not the same as "false", for instance.

My code so far:

import shlex

mystring = 'a=foo, b=bar, c="foo, bar", d=false, e="false"'

splitter = shlex.shlex(mystring, posix=True)
splitter.whitespace += ','
splitter.whitespace_split = True
print list(splitter) # ['a=foo', 'b=bar', 'c=foo, bar', 'd=false', 'e=false']

I believe this is a duplicate of http://stackoverflow.com/a/79985/1540600, which provides a superior answer. (Use `shlex.split()`.) — sferencik, Dec 02 '15 at 12:28
this is not a duplicate, as `shlex.split` can't specify delimitor. — RNA, Feb 22 '16 at 20:27
I agree with RNA. For using a custom delimiter see https://stackoverflow.com/questions/56791723/split-a-string-with-custom-delimiter-respect-and-preserve-quotes-single-or-dou/ — luca, Jun 29 '19 at 11:41

Janne Karila · Accepted Answer · 2016-10-04T09:31:49.127

26

>>> s = r'a=foo, b=bar, c="foo, bar", d=false, e="false", f="foo\", bar"'
>>> re.findall(r'(?:[^\s,"]|"(?:\\.|[^"])*")+', s)
['a=foo', 'b=bar', 'c="foo, bar"', 'd=false', 'e="false"', 'f="foo\\", bar"']

The regex pattern "[^"]*" matches a simple quoted string.
"(?:\\.|[^"])*" matches a quoted string and skips over escaped quotes because \\. consumes two characters: a backslash and any character.
[^\s,"] matches a non-delimiter.
Combining patterns 2 and 3 inside (?: | )+ matches a sequence of non-delimiters and quoted strings, which is the desired result.

edited Oct 04 '16 at 09:31

answered May 23 '13 at 09:55

Janne Karila

22,762
5
50
92

Thank you, works like a charm! – Remo May 23 '13 at 09:57
3

explain how it works and I'll upvote – Jason S Oct 06 '15 at 19:38
1

@JasonS I agree, but [regex101.com](https://regex101.com/) has a useful explanation section that may help, e.g.: https://i.imgur.com/SJMAQPI.png – blah238 Oct 03 '16 at 22:08
that's useful, but that's not the point, the point is that a good answer explains why it works. – Jason S Oct 04 '16 at 02:14
2

@JasonS Added some explanation. – Janne Karila Oct 04 '16 at 12:57
@JanneKarila But this is ignoring the null strings too. How can I retain those? – Muhammad Zeeshan Jun 05 '17 at 03:39
Strictly, this answers the question, but beware that escaping the quote character does not work perfectly for the above regex. An input like `a=\"foo, b="bar"` would not be parsed correctly. – Zorawar Jan 06 '19 at 21:13
@Zorawar Good remark. The encoder I imagined would escape the quote when it adds the quotes around, so that input would be `a="\"foo", b="bar"` – Janne Karila Jan 08 '19 at 06:29

score 0 · Answer 2 · answered May 23 '13 at 09:57

0

Regex can solve this easily enough:

import re

mystring = 'a=foo, b=bar, c="foo, bar", d=false, e="false"'

splitString = re.split(',?\s(?=\w+=)',mystring)

The regex pattern here looks for a whitespace followed by a word character and then an equals sign which splits your string as you desire and maintains any quotes.

answered May 23 '13 at 09:57

ydaetskcoR

47,133
8
134
157

1

This would split `'c="foo, bar="'` – Janne Karila May 23 '13 at 10:04
Fair point. I guess that's the problem with regex, writing something less explicit always seems to catch you out with unexpected cases and yet explicit regex can look horrible to read and understand what's even going on. – ydaetskcoR May 23 '13 at 10:07

Python: Split a string, respect and preserve quotes

2 Answers2

Linked