Pandas: How do I assign values based on multiple conditions for existing columns?

Question

I would like to create a new column with a numerical value based on the following conditions:

a. if gender is male & pet1=pet2, points = 5

b. if gender is female & (pet1 is 'cat' or pet1='dog'), points = 5

c. all other combinations, points = 0

    gender    pet1      pet2
0   male      dog       dog
1   male      cat       cat
2   male      dog       cat
3   female    cat       squirrel
4   female    dog       dog
5   female    squirrel  cat
6   squirrel  dog       cat

I would like the end result to be as follows:

    gender    pet1      pet2      points
0   male      dog       dog       5
1   male      cat       cat       5
2   male      dog       cat       0
3   female    cat       squirrel  5
4   female    dog       dog       5
5   female    squirrel  cat       0
6   squirrel  dog       cat       0

How do I accomplish this?

Please don't repost: http://stackoverflow.com/questions/30630349/pandas-newbie-how-do-i-assign-values-based-on-content-of-columns — EdChum, Jun 03 '15 at 22:39
I split the two part question into two posts and this part has not been answered. — Eric, Jun 03 '15 at 22:43
Fair enough, I didn't see that your other question was updated so have reopened this — EdChum, Jun 03 '15 at 22:45
In your desired result you say that female cat and squirrel should get 5 points, but in your rules you specified in the beginning of the post, this should clearly get 0. What is correct? — firelynx, Jun 04 '15 at 10:17
b. if gender is female & (pet1 is 'cat' or pet1='dog'), points = 5; I apologize if I didn't word this clearly — Eric, Jun 04 '15 at 16:19

Erfan · Answer 1 · 2021-09-30T07:40:19.427

54

`numpy.select`

This is a perfect case for np.select where we can create a column based on multiple conditions and it's a readable method when there are more conditions:

conditions = [
    df['gender'].eq('male') & df['pet1'].eq(df['pet2']),
    df['gender'].eq('female') & df['pet1'].isin(['cat', 'dog'])
]

choices = [5,5]

df['points'] = np.select(conditions, choices, default=0)

print(df)
     gender      pet1      pet2  points
0      male       dog       dog       5
1      male       cat       cat       5
2      male       dog       cat       0
3    female       cat  squirrel       5
4    female       dog       dog       5
5    female  squirrel       cat       0
6  squirrel       dog       cat       0

edited Sep 30 '21 at 07:40

answered Feb 16 '20 at 01:52

Erfan

36,997
6
53
68

3

This is both beautiful and FAST! It handily beats the .apply methods for large data frames. Thank you Erfan, – veg2020 Aug 25 '20 at 21:06
This is very useful for more than 2 conditions. Thanks! – haneulkim Sep 18 '20 at 01:30
8

Note that if you are manipulating an exiting column instead, and if want to keep the existing column value as default you would do: `df['points'] = np.select(conditions, choices, default=df['points'])` more here: https://stackoverflow.com/a/65213788/3750838 – Gera Zenobi Dec 09 '20 at 09:31

score 48 · Accepted Answer · answered Jun 03 '15 at 23:04

You can do this using np.where, the conditions use bitwise & and | for and and or with parentheses around the multiple conditions due to operator precedence. So where the condition is true 5 is returned and 0 otherwise:

In [29]:
df['points'] = np.where( ( (df['gender'] == 'male') & (df['pet1'] == df['pet2'] ) ) | ( (df['gender'] == 'female') & (df['pet1'].isin(['cat','dog'] ) ) ), 5, 0)
df

Out[29]:
     gender      pet1      pet2  points
0      male       dog       dog       5
1      male       cat       cat       5
2      male       dog       cat       0
3    female       cat  squirrel       5
4    female       dog       dog       5
5    female  squirrel       cat       0
6  squirrel       dog       cat       0

score 22 · Answer 3 · answered Jun 03 '15 at 22:54

22

using apply.

def f(x):
  if x['gender'] == 'male' and x['pet1'] == x['pet2']: return 5
  elif x['gender'] == 'female' and (x['pet1'] == 'cat' or x['pet1'] == 'dog'): return 5
  else: return 0

data['points'] = data.apply(f, axis=1)

answered Jun 03 '15 at 22:54

Ruggero Turra

15,540
14
84
131

This works for small DataFrames. When you have more than 1,000,000 rows, it could be very long. – Catalina Chircu Feb 19 '20 at 17:57
Why use this over any of the other methods? – AMC Mar 09 '20 at 19:58
Yes the numpy.select method was much faster for me but this is programmatically elegant. Not sure why this is so slow for a million+ rows dataframe (70 columns) – veg2020 Aug 25 '20 at 21:29

leonard · Answer 4 · 2018-05-14T21:57:55.503

The apply method described by @RuggeroTurra takes a lot longer for 500k rows. I ended up using something like

df['result'] = ((df.a == 0) & (df.b != 1)).astype(int) * 2 + \
               ((df.a != 0) & (df.b != 1)).astype(int) * 3 + \
               ((df.a == 0) & (df.b == 1)).astype(int) * 4 + \
               ((df.a != 0) & (df.b == 1)).astype(int) * 5

where the apply method took 25 seconds and this method above took about 18ms.

score 5 · Answer 5 · answered May 11 '20 at 12:50

You can also use the apply function. For example:

def myfunc(gender, pet1, pet2):
    if gender=='male' and pet1==pet2:
        myvalue=5
    elif gender=='female' and (pet1=='cat' or pet1=='dog'):
        myvalue=5
    else:
        myvalue=0
    return myvalue

And then using the apply function by setting axis=1

df['points'] = df.apply(lambda x: myfunc(x['gender'], x['pet1'], x['pet2']), axis=1)

We get:

     gender      pet1      pet2  points
0      male       dog       dog       5
1      male       cat       cat       5
2      male       dog       cat       0
3    female       cat  squirrel       5
4    female       dog       dog       5
5    female  squirrel       cat       0
6  squirrel       dog       cat       0

score 0 · Answer 6 · answered Mar 24 '22 at 10:46

One option is with case_when from pyjanitor; under the hood it uses pd.Series.mask.

The basic idea is a pairing of condition and expected value; you can pass as many pairings as required, followed by a default value and a target column name:

# pip install pyjanitor
import pandas as pd
import janitor
df.case_when(
    # condition, value
    df.gender.eq('male') & df.pet1.eq(df.pet2), 5,
    df.gender.eq('female') & df.pet1.isin(['cat', 'dog']), 5,
    0, # default
    column_name = 'points')

     gender      pet1      pet2  points
0      male       dog       dog       5
1      male       cat       cat       5
2      male       dog       cat       0
3    female       cat  squirrel       5
4    female       dog       dog       5
5    female  squirrel       cat       0
6  squirrel       dog       cat       0

You could use strings for the conditions, as long as they can be evaluated by pd.eval on the parent dataframe - note that speed wise, this can be slower for small datasets:

df.case_when(
   "gender == 'male' and pet1 == pet2", 5,
   "gender == 'female' and pet2 == ['cat', 'dog']", 5,
   0,
   column_name = 'points')

     gender      pet1      pet2  points
0      male       dog       dog       5
1      male       cat       cat       5
2      male       dog       cat       0
3    female       cat  squirrel       0
4    female       dog       dog       5
5    female  squirrel       cat       5
6  squirrel       dog       cat       0

Anonymous functions are also possible, which can be handy in chained operations:

df.case_when(
    lambda df: df.gender.eq('male') & df.pet1.eq(df.pet2), 5,
    lambda df: df.gender.eq('female') & df.pet1.isin(['cat', 'dog']), 5,
    0, # default
    column_name = 'points')

     gender      pet1      pet2  points
0      male       dog       dog       5
1      male       cat       cat       5
2      male       dog       cat       0
3    female       cat  squirrel       5
4    female       dog       dog       5
5    female  squirrel       cat       0
6  squirrel       dog       cat       0

Pandas: How do I assign values based on multiple conditions for existing columns?

6 Answers6

`numpy.select`

Linked

Related