Calculating time difference between two rows

Question

I'm trying to calculate the time difference between two rows using shift(), but I get an unexpected error. I may be missing something obvious

df['Delta'] = (df.index - df.index.shift(1))

This statement produces a ValueError: Cannot shift with no offset. What am I missing?

Answer 1

Two things:

• If you have a DatetimeIndex, the shift shifts your data with a period of time. If your index has no frequency, you have to provide that to the shift method with the freq keyword (eg freq='s' to shift the data one second)
• You cannot substract two index objects like that, as this gives you a difference set operation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#set-operations-on-index-objects

If you just want the difference between two consecutive values in the index, you can use the diff method (of a Series, a bit easier than shift and substract):

df['index_col'] = df.index
df['Delta'] = df['index_col'].diff()

Answer 2

Perhaps confusingly, pre-1.0 Series.shift and Index.shift used to not exactly do the same thing, the latter only being meaningfully defined for TimesSeries. Probably easiest to add your index as a column.

df['index_col'] = df.index
df['Delta']=(df['index_col'] - df['index_col'].shift(1))