Remove noise from linear signal

Question

This signal represents the distance in an experimental device. As the speed is constant, the signal should be linear (untill it drops). However, for some reason the device showed interferences as shown in the figure below:

The question is how can I remove this high amplitude interferences, making the signal as linear as possible?

I've already tried the butterworth filter and read something about the Chebyshev, but I don't think it applies.

Any suggestions?

@LaurentDuval This is only the beginning of the file, represented by the horizontal line near zero in thebeginning of the plot. Note that this file has 512000 values. — Paulo MiraMor, Jun 21 '16 at 20:36

M529 · Accepted Answer · 2016-06-24T21:19:38.447

5

There are a few simple things that you could try:

sliding-window median filtering with an appropriate window size
linear-regression in a sliding window
robust linear regression to the total signal, if you know the signal duration etc well enough

EDIT: Further idea after a comment from OP on another answer

Since you want the slope at 160000 be the same as before (i.e. no flat section in your signal), you could use a robust linear regression technique such as RANSAC with an appropriate number of sample points to analyze/filter your data.

edited Jun 24 '16 at 21:19

answered Jun 21 '16 at 14:02

M529

1,736
10
15

2

Especially I would like to mention bout the nonlinearity of the required filter. Some nonlinear algorithms such as a variation of median filtering is necessary. A linear filter will have quite difficulty in eliminating spike type noise which has flat spectral distribution over the whole signal spectrum. – Fat32 Jun 21 '16 at 16:39
2

Yes, that is very correct. Thanks, I will remove the Fourier-suggestion since it really does not perform in this case. – M529 Jun 21 '16 at 18:39

score 3 · Answer 2 · edited Apr 13 '17 at 12:47

Non-linearity or non-least-energy seems a key, as stated by @Fat32.

If you want to denoise first, a sliding weighted median (with positive weights) could help you remove impulse noise, while keeping a more accurate location of transitions than a pure median. A Matlab implementation here. If this does not suffice, I'll go for more general rank filters. Median is sort of rank $1/2$. If the impulse noise is too heavy, a $\max$ or a $\min$ filter (toward rank $1$ or $0$) may better follow the slope.

If around $160000$ you want to go through following the apparent (local) slope, then a weighted-median derivative, or a sliding robust linear regression (a robust LOWESS) could produce local derivatives without prior filtering.

Around 160000 I'm expecting that the curve follows the pattern, that is, a diagonal line at the same slope as before. — Paulo MiraMor, Jun 21 '16 at 20:55

Remove noise from linear signal

2 Answers2