Can I use a permutation test on timeseries data?

Question

What I am trying to do:

I am currently doing analysis on neuronal calcium imaging data.

In particular, I have two things:

1. A time series that represents the amount of calcium within a neuron
2. A boolean time series that encodes whether an activity from a mouse is taking place

I want to see if a specific neuron is activated when the defined action is taking place.

The method I want to use:

One technique I read in various paper consists in building a linear classifier on the calcium time series, converting it into a boolean array (1 if it is above the threshold, 0 if it is below). Then this boolean array from calcium is compared to the boolean array that encodes the activity of interest, computing a confusion matrix. This is done to see if the elevation in calcium concentration encodes for the activity.

In particular, we span all the possible thresholds (from a minimum to a maximum) for the calcium imaging data. From the various confusion matrices we can then build a ROC curve and use its area as a performance metric for that particular neuron.

The problem:

In the various papers they then wanted to see whether the results were statistical significant or if they were obtained by pure chance. They tested the significance by circularly permuting the calcium time series (they select a random index "i" of the timeseries, and inverted the timeserie before "i" with the timeseries after "i"). They claim to do the permutation in this way to better preserve the physiological structure of the timeseries.

The thing I do not understand is why this permutation test is applicable to timeseries data? I read about the exchangeability hypothesis that needs to be satisfied before applying this permutation method, but this to me does not seem to be the case... indeed the calcium sample after highly depends on the calcium sample before it. And even if we restrict our analysis to do a circular permutation we have a discontinuity point in the middle...

Questions:

• Is this analysis doable or is the exchangeability hypothesis holding this analysis back?
• If this analysis cannot be done, are there alternative ways to tests the significance if I do not know the underlying null distribution?

Reference article

https://pubmed.ncbi.nlm.nih.gov/31230711/

See “Analysis of Single Cell Responses During Behavior”

Answer 1

The individual-neuron calcium signals described in the Kingsbury et al. paper were evaluated "during behavior events" (maybe better described as behavior epochs) within mice. This was done in by placing two mice to face each other in a tube, and determining periods showing behaviors of "push," "retreat" and "approach" for each mouse over time.

A continuous fluorescent measure of calcium concentration $(\Delta F/F)$ was evaluated across multiple "behavior events" with what seems to have been a standard AUROC (area under the receiver operating characteristic curve) approach. That was evidently done individually for each type of behavior event, over a series of different types of events and periods without the evaluated behaviors.

The circular permutations were done to estimate a random null distribution against which to evaluate the AUROC values within an individual neuron, to identify cells whose activity was associated with a behavior. 1000 circular permutations were done for each neuron.

A neuron was considered significantly responsive (⍺ = 0.05) if its auROC value exceeded the 95th percentile of the random distribution (auROC < 2.5th percentile for suppressed responses, auROC > 97.5th percentile for excited responses).

What's important in that application is to randomize the calcium signals among the behavior types and periods without the specified behaviors. I don't see that the (admittedly large) short-term correlations in the calcium signal over time would pose a problem, provided that the signals were adequately randomized among behaviors/lack-of-behavior. If you are considering a similar approach, that larger-scale randomization is what's critical.

Figure 5 shows that this approach was successful in identifying subsets of cells associated with increased or decreased activity for each of the 3 behaviors. After that classification into cell types based on individual AUROC values, further evaluation was based on continuous measures. For example:

For comparison of response characteristics across subject and opponent cells (Figures S8B and S8C), the response strength for each neuron and each behavior was calculated as the average z-scored $\Delta F/F$ activity during all behavior epochs of a given type. Response probability for each neuron and each behavior was calculated as the percentage of behavior events with average neural activity that exceeded 110% of the local baseline (increased by more than 10% above baseline), taken over the 10 s preceding behavior onset.

Those evaluations further supported the AUROC-based classifications of individual neurons into behavior-associated groups.

Addendum on exchangeability

The lack of exchangeability typical of time-series data doesn't invalidate this method, because of the random variables that need to be considered "exchangeable" here. From Wikipedia, an exchangeable sequence of random variables is one

whose joint probability distribution does not change when the positions in the sequence in which finitely many of them appear are altered.

In this application, what's important is the exchangeability of the observations among behavioral epochs over which evaluations are made: the periods of "push," "retreat," "approach," and "none" identified from mouse behavior. Within any of those individual epochs, the calcium measurements are certainly not exchangeable in time. What's needs for this type of study, however, is that the random variables estimated from the epochs (e.g., the mean $\Delta F/F$ values within individual epochs) are exchangeable. That's over a much different time scale.

For the type of exchangeability needed here, the joint distribution of $\Delta F/F$ values among epochs in a sequence of "approach, none, push" shouldn't be different than it would be if the order were instead "push, approach, none," for example. The structure within each epoch doesn't matter per se. Insofar as that's the case, then the circular permutation to serve as a null for $\Delta F/F$ values within epochs is valid. The circular permutation further maintains the short-term characteristics of the calcium time series even as the mapping between that time series and the behaviors is permuted, removing a potential problem with complete randomization of calcium observations.

---

Kingsbury et al., Correlated Neural Activity and Encoding of Behavior across Brains of Socially Interacting Animals, Cell 178: 429–446 (2019).