I have a huge dataframe which has values and blanks/NA's in it. I want to remove the blanks from the dataframe and move the next values up in the column. Consider below sample dataframe.
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(5,4))
df.iloc[1,2] = np.NaN
df.iloc[0,1] = np.NaN
df.iloc[2,1] = np.NaN
df.iloc[2,0] = np.NaN
df
0 1 2 3
0 1.857476 NaN -0.462941 -0.600606
1 0.000267 -0.540645 NaN 0.492480
2 NaN NaN -0.803889 0.527973
3 0.566922 0.036393 -1.584926 2.278294
4 -0.243182 -0.221294 1.403478 1.574097
I want my output to be as below
0 1 2 3
0 1.857476 -0.462941 -0.600606 NaN
1 0.000267 -0.540645 0.492480 NaN
2 -0.803889 0.527973 NaN NaN
3 0.566922 0.036393 -1.584926 2.278294
4 -0.243182 -0.221294 1.403478 1.574097
df.shift(-1,axis=1) is the concept I want, but shifts even non-nan values which is not what i want. I saw the answer to shift it up with df[cols].apply(lambda x: pd.Series(x.dropna().values))
I want to drop the NaNs from the row and move non-NaN values to the left most part.