43

I use the following method a lot to append a single row to a dataframe. One thing I really like about it is that it allows you to append a simple dict object. For example:

# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])

# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)

Again, what I like most about this is that the code is very clean and requires very few lines. Now I suppose the recommended alternative is:

# Create the new row as its own dataframe
df_new_row = pd.DataFrame({ 'a': [1], 'b': [2] })
df = pd.concat([df, df_new_row])

So what was one line of code before is now two lines with a throwaway variable and extra cruft where I create the new dataframe. :( Is there a good way to do this that just uses a dict like I have in the past (that is not deprecated)?

Glenn
  • 2,982
  • 8
  • 31
  • 37
  • 5
    [pandas issue 35407](https://github.com/pandas-dev/pandas/issues/35407) explains that `df.append` was deprecated because: "Series.append and DataFrame.append [are] making an analogy to list.append, but it's a poor analogy since the behavior isn't (and can't be) in place. The data for the index and values needs to be copied to create the result." – Paul Rougieux Feb 10 '22 at 12:15
  • 2
    Came across this warning today. However when I used concat as the alternative I got "cannot concatenate object of type ''; only Series and DataFrame objs are valid". So frustrating..... – Ben Watson Feb 13 '22 at 23:16

3 Answers3

24

Create a list with your dictionaries, if they are needed, and then create a new dataframe with df = pd.DataFrame.from_records(your_list). List's "append" method are very efficient and won't be ever deprecated. Dataframes on the other hand, frequently have to be recreated and all data copied over on appends, due to their design - that is why they deprecated the method

jsbueno
  • 86,446
  • 9
  • 131
  • 182
  • 1
    How do you know that it is deprecated? At https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.append.html#pandas.DataFrame.append (which currently shows version 1.4.0) I don't see any mention about that. Even at the dev tree I don't see any deprecation warning: https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.append.html – zby Feb 02 '22 at 10:55
  • 3
    I agree ; though when you use append method (with 1.4.0) you run into a "FutureWarning : The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead". You will find the details in the ["what's new" page](https://pandas.pydata.org/docs/dev/whatsnew/v1.4.0.html#deprecated-frame-append-and-series-append) – tgrandje Feb 04 '22 at 13:20
  • 1
    @zby the update to the documentation is being dealt with in this pull request: https://github.com/pandas-dev/pandas/pull/45587 – Paul Rougieux Feb 10 '22 at 12:21
  • this brought a ten times faster speed to my code , thanks so much man – kush May 07 '22 at 13:37
  • That is actually the reason they are deprecating `df.append`. Thank the Pandas maintainers for that. Still, the "new way to do it" should be more proeminent in their docs, for sure. – jsbueno May 07 '22 at 16:50
5

I also like the append method. But you can do it in one line with a list of dicts

df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])])

or using loc and tuples for values on DataFrames with incremenal ascending indexes

df.loc[len(df), ['a','b']] = 1, 2

or maybe

df.loc[len(df), df.columns] = 3, 4
Rafael Gaitan
  • 51
  • 4
  • 1
    You can also use ignore_index `df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])], ignore_index=True)` – Rafael Gaitan Apr 22 '22 at 18:50
1

This have already good answare, but I just wanted to append something at the end here as I didnt find it :) If you are appending a DF to a DF and want to use concat this is the way I found:

original with append

   outputxlsx = outputxlsx.append( df, ignore_index=True)

concat

   outputxlsx = pd.concat([outputxlsx, pd.DataFrame.from_records(df)])
beltalowda
  • 11
  • 1