3

I'm using read_csv to read a fairly big csv in chunks (just reading the first line to test).

data = read_csv('VOD_Properties.csv', nrows=1, low_memory=False)
print(data)

Result:

                                      PROPERTIES
0  {"video_id":241338,"play_uuid":"0d293b16-566a-...

Original data in excel:

{"video_id":241338,"play_uuid":"0d293b16-566a-46e7-ac90-e3caa602a527","seconds":0.116,"current_state":"PLAY","total_seconds":100.032}

To test if the data's just simply not showing up, I converted into a string:

string_data = data.to_string()

and print the last few characters of the string to see if they're '...':

6-566a-...

I've tried playing around with the parameters but not a single improvement.

Thanks in advance.

smci
  • 29,564
  • 18
  • 109
  • 144
T.Y
  • 33
  • 1
  • 5
  • I don't think it's an issue of reading all the data, but rather an issue of not displaying all of it. Numpy, Pandas, etc., use ellipsis to indicate not all data was shown. – Ami Tavory Sep 22 '16 at 20:32
  • 1
    The line was read fine; just when printing pandas truncates wide columns with ellipsis '...'. You can change pandas' column width display threshold with: [How to remove ellipsis from a row in a Python Pandas series or data frame?](http://stackoverflow.com/questions/21028819/how-to-remove-ellipsis-from-a-row-in-a-python-pandas-series-or-data-frame) Should we close this as a duplicate of that? – smci Sep 22 '16 at 20:34
  • 1
    Dupe: http://stackoverflow.com/questions/26277757/pandas-to-html-truncates-string-contents and http://stackoverflow.com/questions/21028819/how-to-remove-ellipsis-from-a-row-in-a-python-pandas-series-or-data-frame – EdChum Sep 22 '16 at 20:35
  • T.Y., I already posted your solution: `pandas.set_option('display.max_colwidth', 1000)` – smci Sep 22 '16 at 20:37
  • smci, yes works perfectly. Thank you! – T.Y Sep 22 '16 at 20:42

1 Answers1

4

The full line was in fact read correctly, just when printing pandas truncates wide columns with ellipsis '...'.

You can change pandas' column width display threshold as per: How to remove ellipsis from a row in a Python Pandas series or data frame, shown when long lines/wide columns are truncated?

pandas.set_option('display.max_colwidth', 1000) # or whatever width
Community
  • 1
  • 1
smci
  • 29,564
  • 18
  • 109
  • 144