Cross Validation in application of clustering on a collection of similarly behaving time series

Question

I'm trying to understand how and at which point can one apply Cross Validation for time series data. If i'm not wrong CV increases generalisation so that our model has less bias in case the data is ordered/distributed in a certain particular way. But time series inherently has a direction (it can only move forward) and if i'm using some unsupervised algorithms on it, how should one apply Cross validation on it if any. I read a related question on it here : (Cross-validation techniques for time series data ) although as opposed to the question above for supervised learning i'm planning on some unsupervised clustering algorithms to group a set of time series together. So basically i have two doubts :

How to approach the problem of CV for my time series problem.
majority of questions related to cv and timeseries and cv used supervised algorithms? So an insight taking into account the clustering technique i would be using would be preferable.

score 1 · Answer 1 · answered Sep 02 '18 at 13:56

If i'm not wrong CV increases generalisation so that our model has less bias in case the data is ordered/distributed in a certain particular way.

No, your understanding is wrong here. CV in itself does not help with generalization, it can only help in measuring the generalization abilities of a model.

And even that is only possible if you have a suitable figure of merit that is evaluated. Such figures of merit that measure the performance of predictions are readily availbe for supervised models. For unsupervised models such as cluster analysis it is much more difficult to formulate them. Models can differ in ways that are unimportant to the generalization ability and/or they can differ in ways that are important, and only the latter differences are of interest to us. You may also say that of all ways how a model can differ, we are only interested in a subset and the question is how to catch that subset.

Predictive models have one easy to capture type of important differences: differences in the prediction, and the usual figures of merit track exactly those differences. For your cluster analysis, you'll have to specify what differences are important and what differences are unimportant/benign. This is something that needs to be done in close relation to the application (data generating processes, characteristics of your data, and the question(s) you want to answer).

Using a clustering technique pretty much implies that you do not have a ground truth to compare the clustering with. This is what limits the general possibility of having figures of merit similar to the ones we have for predicitive models. However, if you do have something like a ground truth, you may derive a figure of merit based on that ground truth.

Ensemble models can help with generalization in certain situations and they are linked to resampling validation (such as CV) in that they also use a resampling step.

Time series cross validation: one crucial question that makes a lot of difference in how to set up the splitting is whether you are looking along time series.

For prediction that is training on past events and then predict future developments of the same time series. An example in process analysis would be to train a model predicting current (or future) concentration based on past readings of some sensors.
Or, whether you look across a large number of time series, e.g. series of sensor readings for past runs of a batch process against the achieved final product concentrations and then want to predict final product concentration for unknown batches based on the respective full time series of sensor readings for the batch in question.

Sliding window cross validation is for the first scenario, the 2nd scenario uses each time series as its own case and then "normal" cross validation.

Correlation and stack of time series. The crucial question here is not whether there's a similarity in the time series but what causes the similarities. If there's an underlying factor causing similarity (i.e. you have several pools of servers in your data and the server data is more similar with each pool than compared to a server of another pool), you need to take that into account for CV splitting. If on the other hand the similarity is not caused by other factors but is just the phenomenon you're modeling, that is fine.

Thanks cbeleites for your answer and particularly about explaining about what CV really does. You are right i don't have any ground truths and that's why i went with the clustering approach, and the only metric that i can seem to use right out of the box is silhouette score. I understood your approach of determining the differences that really matter and then maybe using them for improving on our base algorithms. — Mr. Confused, Sep 04 '18 at 05:38
Also my use case here is not to predict some future value but just study the behaviour of lets say 5 servers (lets say their memory utilisation in 1 min ticks) running together and then find whether any of them is behaving abnormally by clustering the similar behaviours together and others not in that group. sliding window CV i get but what is normal "CV". Is it like ensemble average i.e. CV across multiple signals? Should we do that and how to do that ? I know it's asking too much but can you provide an example or a more complete description. It will be a lot of help. Thanks. — Mr. Confused, Sep 04 '18 at 05:38

Cross Validation in application of clustering on a collection of similarly behaving time series

1 Answers1