This question may be too basic. For a temporal trend of a data, I would like to find out the point where "abrupt" change happens. For example, in the first figure shown below, I would like to find out the change point using some statistic method. And I would like to apply such method in some other data of which the change point is not obvious (like the 2nd figure).So is there a common method for such purpose?
If the observations of your time series data are correlated with the immediately previous observations, the paper by Chen and Liu (1993)$^{[1]}$ may interest you. It describes a method to detect level shifts and temporary changes in the framework of autoregressive moving-average time series models.
[1]: Chen, C. and Liu, L-M. (1993),
"Joint Estimation of Model Parameters and Outlier Effects in Time Series,"
Journal of the American Statistical Association, 88:421, 284-297
This problem in Stats is referred to as the (univariate) Temporal Event Detection. The simplest idea is to use a moving average and standard deviation. Any reading that is "out of" 3-standard deviations (rule-of-thumb) is considered an "event". There of course, are more advanced models that use HMMs, or Regression. Here is an introductory overview of the field.
This inference problem has many names, including change points, switch points, break points, broken line regression, broken stick regression, bilinear regression, piecewise linear regression, local linear regression, segmented regression, and discontinuity models.
Here is an overview of change point packages with pros/cons and worked examples. If you know the number of change points a priori, check out the mcp package. First, let's simulate the data:
df = data.frame(x = seq(1, 12, by = 0.1))
df$y = c(rnorm(21, 0, 5), rnorm(80, 180, 5), rnorm(10, 20, 5))
For your first problem, it's three intercept-only segments:
model = list(
y ~ 1, # Intercept
~ 1, # etc...
~ 1
)
library(mcp)
fit = mcp(model, df, par_x = "x")
We can plot the resulting fit:
plot(fit)
Here, the change points are very well defined (narrow). Let's summarise the fit to see their inferred locations (cp_1 and cp_2):
summary(fit)
Family: gaussian(link = 'identity')
Iterations: 9000 from 3 chains.
Segments:
1: y ~ 1
2: y ~ 1 ~ 1
3: y ~ 1 ~ 1
Population-level parameters:
name mean lower upper Rhat n.eff
cp_1 3.05 3.0 3.1 1 6445
cp_2 11.05 11.0 11.1 1 6401
int_1 0.14 -1.9 2.1 1 5979
int_2 179.86 178.8 180.9 1 6659
int_3 22.76 19.8 25.5 1 5906
sigma_1 4.68 4.1 5.3 1 5282
You can do much more complicated models with mcp, including modeling Nth-order autoregression (useful for time series), etc. Disclosure: I am the developer of mcp.
The area of statistics that you are looking for is changepoint analysis. There is a website here that will give you an overview of the area and also have a page for software.
If you are an R user then i'd recommend the changepoint package for changes in mean and the strucchange package for changes in regression. If you want to be Bayesian then the bcp package is good too.
In general you have to choose a threshold which indicates the strength of the changes you are looking for. There are, of course, threshold choices that people advocate in certain situations and you can use asymptotic confidence levels or bootstrapping to get confidence too.
Try roerich library for abrupt change detection in time series. It contains several algorithms for change point detection. Each of them returns a score that you can use to characterize the detected changes.
Example on a simple case with several abrupt changes:
import numpy as np
# generate a sample
n = 100
X1 = np.random.normal(0, 0.1, size=(n, 1))
X2 = np.random.normal(1, 0.1, size=(n, 1))
X3 = np.random.normal(0, 0.1, size=(n, 1))
X = np.concatenate((X1, X2, X3), axis=0)
cps_true = [n, 2*n]
Now, apply one of the algorithms in the library:
import roerich
from roerich.change_point import SlidingWindows
change points detection
cpd = SlidingWindows(window_size=20, metric='energy', bootstrap=False, n_runs=1)
score, cps_pred = cpd.predict(X)
visualization
roerich.display(X, cps_true, score, cps_pred=None)
Result:
More examples are in this short intro on Medium.
Here is a quick and easy way to do it. Create a bunch of jump functions like this: $$ J_i = \left\{\begin{array}{l@{\qquad}l} 0 & x < x_i\\ 1 & x \ge x_i \end{array}\right. $$ for candidate cutoff points $x_1<x_2<\cdots<x_m$. Now use stepwise regression to select the best model with the $J_i$ as possible predictors. In your first example, assuming you select two predictors, you'll get one for $J_{april}$ with a positive coefficient equal to the size of the jump upward, and one for $J_{december}$ with a negative coefficient equal to the size of the jump downward. You need to decide how finely you want to divide the candidate jump times, $x_i$, e.g., one per month, one per fortnight, one per week, one per day.
There are more elegant and exacting solutions involving nonlinear regression, where you use a model with $J_1$ and $J_2$ and estimate $x_1$ and $x_2$ as parameters. It's a bit messy to set up.
There is a related problem of dividing a series or sequence into spells with ideally constant values. See How can I group numerical data into naturally forming "brackets"? (e.g. income)
It's not quite the same problem as the question doesn't exclude spells with slow drift in any or all directions, but without abrupt changes.
A more direct answer is to say that we are looking for big jumps, so the only real issue is to define jump. The first idea is then just to look at first differences between neighbouring values. It's not even clear that you need to refine that by removing noise first, as if jumps can't be distinguished from differences in noise, they surely can't be abrupt. On the other hand, the questioner evidently wants abrupt change to include ramped as well as stepped change, so some criterion such as variance or range within fixed-length windows seems called for.
