Reindex 2nd level in multi-level dataframe

Question

I need to reindex the 2nd level of a pandas dataframe, so that the 2nd level becomes a list np.arange(N) for each 1st level index. I tried to follow this, but unfortunately it only creates an index with as many rows as previously existing. What I want is that for each new index new rows are inserted (with nan values).

In [79]:

df = pd.DataFrame({
  'first': ['one', 'one', 'one', 'two', 'two', 'three'], 
  'second': [0, 1, 2, 0, 1, 1],
  'value': [1, 2, 3, 4, 5, 6]
})
print df
   first  second  value
0    one       0      1
1    one       1      2
2    one       2      3
3    two       0      4
4    two       1      5
5  three       1      6
In [80]:

df['second'] = df.reset_index().groupby(['first']).cumcount()
print df
   first  second  value
0    one       0      1
1    one       1      2
2    one       2      3
3    two       0      4
4    two       1      5
5  three       0      6

My desired result is:

   first  second  value
0    one       0      1
1    one       1      2
2    one       2      3
3    two       0      4
4    two       1      5
4    two       2      nan
5  three       0      6
5  three       1      nan
5  three       2      nan

Could you just first create the data frame with all of the rows you need. Then update it with the values you have. — Pekka, Aug 09 '15 at 08:22
are the indices in "second" always contiguous and starting from `0`? — chris-sc, Aug 09 '15 at 08:22

score 4 · Accepted Answer · answered Aug 09 '15 at 09:08

I think you can first set columns first and second as multi-level index, and then reindex.

# your data
# ==========================
df = pd.DataFrame({
  'first': ['one', 'one', 'one', 'two', 'two', 'three'], 
  'second': [0, 1, 2, 0, 1, 1],
  'value': [1, 2, 3, 4, 5, 6]
})

df

   first  second  value
0    one       0      1
1    one       1      2
2    one       2      3
3    two       0      4
4    two       1      5
5  three       1      6

# processing
# ============================
multi_index = pd.MultiIndex.from_product([df['first'].unique(), np.arange(3)], names=['first', 'second'])

df.set_index(['first', 'second']).reindex(multi_index).reset_index()

   first  second  value
0    one       0      1
1    one       1      2
2    one       2      3
3    two       0      4
4    two       1      5
5    two       2    NaN
6  three       0    NaN
7  three       1      6
8  three       2    NaN

why is the `.reset_index` required after `.reindex`? – Gulzar Oct 29 '20 at 15:19 — Gulzar, Oct 29 '20 at 15:19

Reindex 2nd level in multi-level dataframe

1 Answers1

Linked