Python - Convert a bytes array into JSON format

Question

I want to parse a bytes string in JSON format to convert it into python objects. This is the source I have:

my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]'

And this is the desired outcome I want to have:

[{
"Date": "2016-05-21T21:35:40Z",
"CreationDate": "2012-05-05",
"LogoType": "png",
"Ref": 164611595,
"Classes": [
  "Email addresses",
  "Passwords"
],
"Link": "http://some_link.com"}]

First, I converted the bytes to string:

my_new_string_value = my_bytes_value.decode("utf-8")

but when I try to invoke loads to parse it as JSON:

my_json = json.loads(my_new_string_value)

I get this error:

json.decoder.JSONDecodeError: Expecting value: line 1 column 174 (char 173)

I've converted the bytes to string by using `.decode("utf-8")` but when I try to convert the string to JOSN i get this error json.decoder.JSONDecodeError: Expecting value:line 1 column 174 (char 173) — Merouane Benthameur, Oct 15 '16 at 13:49
Can you update your question with the relevant code and print out the decoded string? — OneCricketeer, Oct 15 '16 at 13:50
@MerouaneBenthameur The reason it fails is because the string you have **is not** JSON. The most obvious thing is that JSON uses `"`, not `'`. — freakish, Oct 15 '16 at 13:58
Also see [Convert bytes to a string](https://stackoverflow.com/q/40059654/4014959). — PM 2Ring, Jun 22 '20 at 16:41

PM 2Ring · Accepted Answer · 2020-06-22T16:50:57.733

Your bytes object is almost JSON, but it's using single quotes instead of double quotes, and it needs to be a string. So one way to fix it is to decode the bytes to str and replace the quotes. Another option is to use ast.literal_eval; see below for details. If you want to print the result or save it to a file as valid JSON you can load the JSON to a Python list and then dump it out. Eg,

import json

my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]'

# Decode UTF-8 bytes to Unicode, and convert single quotes 
# to double quotes to make it valid JSON
my_json = my_bytes_value.decode('utf8').replace("'", '"')
print(my_json)
print('- ' * 20)

# Load the JSON to a Python list & dump it back out as formatted JSON
data = json.loads(my_json)
s = json.dumps(data, indent=4, sort_keys=True)
print(s)

output

[{"Date": "2016-05-21T21:35:40Z", "CreationDate": "2012-05-05", "LogoType": "png", "Ref": 164611595, "Classe": ["Email addresses", "Passwords"],"Link":"http://some_link.com"}]
- - - - - - - - - - - - - - - - - - - - 
[
    {
        "Classe": [
            "Email addresses",
            "Passwords"
        ],
        "CreationDate": "2012-05-05",
        "Date": "2016-05-21T21:35:40Z",
        "Link": "http://some_link.com",
        "LogoType": "png",
        "Ref": 164611595
    }
]

As Antti Haapala mentions in the comments, we can use ast.literal_eval to convert my_bytes_value to a Python list, once we've decoded it to a string.

from ast import literal_eval
import json

my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]'

data = literal_eval(my_bytes_value.decode('utf8'))
print(data)
print('- ' * 20)

s = json.dumps(data, indent=4, sort_keys=True)
print(s)

Generally, this problem arises because someone has saved data by printing its Python repr instead of using the json module to create proper JSON data. If it's possible, it's better to fix that problem so that proper JSON data is created in the first place.

I don't believe it is a JSON string, rather a Python repr, so use literal_eval instead — Antti Haapala -- Слава Україні, Oct 16 '16 at 06:49
BTW, if you want to analyze or traverse a complicated JSON structure please see https://stackoverflow.com/a/52414034/4014959 & https://stackoverflow.com/a/41778581/4014959 — PM 2Ring, Nov 24 '20 at 02:14
Re: _Generally this problem arises ... proper JSON data is created_: Bytes data is apparently not JSON serializable: `json.dumps(b'\0x41\x45')` -> `TypeError: Object of type bytes is not JSON serializable` — Vercingatorix, Mar 18 '22 at 14:05
@Vercingatorix JSON is for serializing data that's ultimately composed of strings, numbers, and booleans (or null), it's not designed to cope with arbitrary binary data. But it's easy enough to transform such data into a JSON-friendly form, eg you can create a simple hex string with https://docs.python.org/3/library/stdtypes.html#bytes.hex Or you could optimize that slightly with https://docs.python.org/3/library/base64.html — PM 2Ring, Mar 18 '22 at 14:24

score 71 · Answer 2 · answered Oct 16 '20 at 08:11

71

You can simply use,

import json

json.loads(my_bytes_value)

answered Oct 16 '20 at 08:11

Chaithanya Krishna

960
6
12

8

this should have green tick. – kishore Jul 17 '21 at 17:11

score 5 · Answer 3 · answered Dec 16 '20 at 19:45

5

Python 3.5 + Use io module

import json
import io

my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]'

fix_bytes_value = my_bytes_value.replace(b"'", b'"')

my_json = json.load(io.BytesIO(fix_bytes_value))

answered Dec 16 '20 at 19:45

Novikov

184
2
10

1

thank you! this is the the least icky and most direct path, particularly if the json within the bytesarray is already properly formatted. – Jason Rahm Jan 21 '21 at 04:16

score 2 · Answer 4 · answered Oct 15 '16 at 13:55

To convert this bytesarray directly to json, you could first convert the bytesarray to a string with decode(), utf-8 is standard. Change the quotation markers.. The last step is to remove the " from the dumped string, to change the json object from string to list.

dumps(s.decode()).replace("'", '"')[1:-1]

score 1 · Answer 5 · answered Feb 28 '22 at 15:55

Better solution is:

import json
byte_array_example = b'{"text": "\u0627\u06CC\u0646 \u06CC\u06A9 \u0645\u062A\u0646 \u062A\u0633\u062A\u06CC \u0641\u0627\u0631\u0633\u06CC \u0627\u0633\u062A."}'    
res = json.loads(byte_array_example.decode('unicode_escape'))
print(res)

result:

{'text': 'این یک متن تستی فارسی است.'}

decode by utf-8 cannot decode unicode characters. The right solution is uicode_escape

It is OK

score 0 · Answer 6 · edited Apr 08 '22 at 11:46

0

d = json.dumps(byte_str.decode('utf-8'))

edited Apr 08 '22 at 11:46

Suraj Rao

28,850
10
94
99

answered Apr 08 '22 at 11:43

Kwame

1,084
2
20
38

Python - Convert a bytes array into JSON format

6 Answers6

Linked

Related