I want to generate a column with random numbers like this:
df=df.withColumn("random_col",random.randint(100000, 1000000))
The above gives me an error:
AssertionError: col should be Column
Asked
Active
Viewed 1,043 times
0
Marcela Bejarano
- 103
- 1
- 1
- 7
-
Nice , so where are you stuck? – anky May 08 '20 at 18:50
-
check https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.rand – murtihash May 08 '20 at 18:58
-
I get the error AssertionError: col should be Column, any ideas? – Marcela Bejarano May 08 '20 at 18:58
-
I don't use dataframes, but it might want a Column object instead of a numpy array as the argument. – TechPerson May 08 '20 at 18:59
-
@TechPerson good point. Any ideas on how to make it a column object? – Marcela Bejarano May 08 '20 at 19:02
-
I'll check the documentation. – TechPerson May 08 '20 at 19:03
1 Answers
0
First I would make sure you have imported the correct stuff...
Try importing: from pyspark.sql.functions import rand
And then trying something like this line of code:
df1 = df.withColumn("random_col", rand() > 100000, 1000000)
You also could check out this resource. It looks like it may be helpful for what you are doing
Hope this helps!
love2phish
- 229
- 1
- 4
- 13