PySpark: when function with multiple outputs

Question

I am trying to use a "chained when" function. In other words, I'd like to get more than two outputs.

I tried using the same logic of the concatenate IF function in Excel:

  df.withColumn("device_id", when(col("device")=="desktop",1)).otherwise(when(col("device")=="mobile",2)).otherwise(null))

But that doesn't work since I can't put a tuple into the "otherwise" function.

score 50 · Accepted Answer · edited Apr 09 '18 at 14:04

50

Have you tried:

from pyspark.sql import functions as F
df.withColumn('device_id', F.when(col('device')=='desktop', 1).when(col('device')=='mobile', 2).otherwise(None))

Note that when chaining when functions you do not need to wrap the successive calls in an otherwise function.

edited Apr 09 '18 at 14:04

pault

37,170
13
92
132

answered Mar 01 '17 at 16:58

Grr

14,506
7
57
78

It works! Thank you very much!! – Fede Mar 03 '17 at 11:30
Super useful the `when` chaining composition! Thanks! – raul ferreira Oct 18 '18 at 15:06
2

These are chained operations - if multiple when() are true, does the variable get assigned the first condition that is True or the last one? – Thomas Nov 18 '18 at 18:55
1

@Thomas If multiple consecutive when() statements are true, only the first when() evaluated to true is considered. – sondrelv Oct 16 '19 at 13:43

PySpark: when function with multiple outputs

1 Answers1

Linked