How do I Pandas group-by to get sum?

Question

I am using this data frame:

Fruit   Date      Name  Number
Apples  10/6/2016 Bob    7
Apples  10/6/2016 Bob    8
Apples  10/6/2016 Mike   9
Apples  10/7/2016 Steve 10
Apples  10/7/2016 Bob    1
Oranges 10/7/2016 Bob    2
Oranges 10/6/2016 Tom   15
Oranges 10/6/2016 Mike  57
Oranges 10/6/2016 Bob   65
Oranges 10/7/2016 Tony   1
Grapes  10/7/2016 Bob    1
Grapes  10/7/2016 Tom   87
Grapes  10/7/2016 Bob   22
Grapes  10/7/2016 Bob   12
Grapes  10/7/2016 Tony  15

I want to aggregate this by Name and then by fruit to get a total number of Fruit per Name. For example:

Bob,Apples,16

I tried grouping by Name and Fruit but how do I get the total number of Fruit?

you can use dfsql df.sql('SELECT fruit, sum(number) GROUP BY fruit') https://github.com/mindsdb/dfsql https://medium.com/riselab/why-every-data-scientist-using-pandas-needs-modin-bringing-sql-to-dataframes-3b216b29a7c0 — Jorge Torres, Apr 20 '21 at 06:35

score 392 · Accepted Answer · edited Apr 06 '20 at 07:59

392

Use GroupBy.sum:

df.groupby(['Fruit','Name']).sum()

Out[31]: 
               Number
Fruit   Name         
Apples  Bob        16
        Mike        9
        Steve      10
Grapes  Bob        35
        Tom        87
        Tony       15
Oranges Bob        67
        Mike       57
        Tom        15
        Tony        1

edited Apr 06 '20 at 07:59

yatu

80,714
11
64
111

answered Oct 07 '16 at 17:37

Steven G

14,602
6
47
72

156

How can pandas knows that I want to sum the col named `Number` ? – Kingname Oct 23 '17 at 12:32
25

@Kingname it's the last column left if you take out NAME and FRUIT. if you add 2 columns left, it would sum both columns – Steven G Oct 23 '17 at 16:51
`Date` is not summed because it has `dtype = string` yes? – Wassadamo Sep 01 '18 at 02:28
36

How to specify which column to sum? – tgdn Nov 05 '19 at 14:38
125

@tgdn df.groupby(['Name', 'Fruit'])['Number'].sum() – Steven G Nov 08 '19 at 17:34
7

@StevenG For the answer provided to sum up a specific column, the output comes out as a Pandas series instead of Dataframe. From the comment by Jakub Kukul (in below answer), we can use double square brackets around 'Number' to get a Dataframe. – skdhfgeq2134 Jan 16 '20 at 10:41
@tgdn .sum('col_name') – Viktor Avdulov Mar 17 '21 at 04:00

score 244 · Answer 2 · answered Oct 08 '16 at 11:40

244

Also you can use agg function,

df.groupby(['Name', 'Fruit'])['Number'].agg('sum')

answered Oct 08 '16 at 11:40

Saurabh

6,863
4
43
42

3

This differs from the accepted answer in that this returns a `Series` whereas the other returns a `GroupBy` object. – Gaurang Tandon May 08 '19 at 15:53
43

@GaurangTandon to get `DataFrame` object instead (like in the accepted answer), use double square brackets around `'Number'`, i.e.: `df.groupby(['Name', 'Fruit'])[['Number']].agg('sum')` – Jakub Kukul Aug 21 '19 at 17:05
1

Very helpful in cleaning up badly-encoded query report. – SearchTools-Avi Oct 09 '19 at 20:39

score 174 · Answer 3 · answered Jul 02 '18 at 10:01

If you want to keep the original columns Fruit and Name, use reset_index(). Otherwise Fruit and Name will become part of the index.

df.groupby(['Fruit','Name'])['Number'].sum().reset_index()

Fruit   Name       Number
Apples  Bob        16
Apples  Mike        9
Apples  Steve      10
Grapes  Bob        35
Grapes  Tom        87
Grapes  Tony       15
Oranges Bob        67
Oranges Mike       57
Oranges Tom        15
Oranges Tony        1

As seen in the other answers:

df.groupby(['Fruit','Name'])['Number'].sum()

               Number
Fruit   Name         
Apples  Bob        16
        Mike        9
        Steve      10
Grapes  Bob        35
        Tom        87
        Tony       15
Oranges Bob        67
        Mike       57
        Tom        15
        Tony        1

score 58 · Answer 4 · answered Oct 07 '16 at 18:35

Both the other answers accomplish what you want.

You can use the pivot functionality to arrange the data in a nice table

df.groupby(['Fruit','Name'],as_index = False).sum().pivot('Fruit','Name').fillna(0)



Name    Bob     Mike    Steve   Tom    Tony
Fruit                   
Apples  16.0    9.0     10.0    0.0     0.0
Grapes  35.0    0.0     0.0     87.0    15.0
Oranges 67.0    57.0    0.0     15.0    1.0

score 30 · Answer 5 · answered Mar 11 '18 at 00:29

30

df.groupby(['Fruit','Name'])['Number'].sum()

You can select different columns to sum numbers.

answered Mar 11 '18 at 00:29

jared

432
4
9

score 20 · Answer 6 · answered Feb 02 '20 at 08:25

A variation on the .agg() function; provides the ability to (1) persist type DataFrame, (2) apply averages, counts, summations, etc. and (3) enables groupby on multiple columns while maintaining legibility.

df.groupby(['att1', 'att2']).agg({'att1': "count", 'att3': "sum",'att4': 'mean'})

using your values...

df.groupby(['Name', 'Fruit']).agg({'Number': "sum"})

score 12 · Answer 7 · answered Nov 21 '18 at 03:01

You can set the groupby column to index then using sum with level

df.set_index(['Fruit','Name']).sum(level=[0,1])
Out[175]: 
               Number
Fruit   Name         
Apples  Bob        16
        Mike        9
        Steve      10
Oranges Bob        67
        Tom        15
        Mike       57
        Tony        1
Grapes  Bob        35
        Tom        87
        Tony       15

score 8 · Answer 8 · answered Mar 18 '21 at 11:52

You could also use transform() on column Number after group by. This operation will calculate the total number in one group with function sum, the result is a series with the same index as original dataframe.

df['Number'] = df.groupby(['Fruit', 'Name'])['Number'].transform('sum')
df = df.drop_duplicates(subset=['Fruit', 'Name']).drop('Date', 1)

Then, you can drop the duplicate rows on column Fruit and Name. Moreover, you can drop the column Date by specifying axis 1 (0 for rows and 1 for columns).

# print(df)

      Fruit   Name  Number
0    Apples    Bob      16
2    Apples   Mike       9
3    Apples  Steve      10
5   Oranges    Bob      67
6   Oranges    Tom      15
7   Oranges   Mike      57
9   Oranges   Tony       1
10   Grapes    Bob      35
11   Grapes    Tom      87
14   Grapes   Tony      15

# You could achieve the same result with functions discussed by others: 
# print(df.groupby(['Fruit', 'Name'], as_index=False)['Number'].sum())
# print(df.groupby(['Fruit', 'Name'], as_index=False)['Number'].agg('sum'))

There is an official tutorial Group by: split-apply-combine talking about what you can do after group by.

This was exactly what I was looking for!! I was lost how to add the ```sum``` value from ```groupby()[].sum()``` to the "unique" dataframe! Thanks! — DGKang, Aug 11 '21 at 14:37

score 1 · Answer 9 · answered Apr 20 '21 at 06:36

You can use dfsql
for your problem, it will look something like:

df.sql('SELECT fruit, sum(number) GROUP BY fruit')

https://github.com/mindsdb/dfsql

here is an article about it:

https://medium.com/riselab/why-every-data-scientist-using-pandas-needs-modin-bringing-sql-to-dataframes-3b216b29a7c0

How do I Pandas group-by to get sum?

9 Answers9

Linked

Related