5

I am interested in looking for periodicities in a several day long recording of electrical activity. The traces present a very steady baseline over which, from time to time, some short events (300-500 ms) appear (hence the sparse in the title, although I am not sure it is the right term to use).

Now, the nature of the data prevents me to analyse the trace as a whole (~20 days recorded at 10KHz, my computer will definitely not handle that), so I just wrote a series of routines to find the interesting events, and export their date/time (also their shape, but that is another story) to a text file.

So, I end up with a series of times to analyse.

Just by looking at the raw trace what I can see is:

  1. A ~2 hours periodicity in the events
  2. The events appear in groups. That is, I have a series of 20-30 events in the course of a couple of minutes, then nothing for ~2 hours, then other 20-30 events and so on. Note that the number of events in a group is variable, can be 30 in one and 5 in the next one.
  3. The periodicity is fairly obvious, but it's not perfect: events happen roughly at 2 hours interval, and the exact time may vary from day to day.
  4. There may be superimposed periodicities. In particular, I see events at a specific time of the day, which may or may not fit in the abovementioned 2 hours periodicity.

What can I do to statistically determine the periodicity?

What I have tried to do is to make a frequency histogram of the events, binning every 10 minutes, which empirically seemed like a sensible bin size, and then looking at the ACF or the FFT of the counts, but only in a few cases something pops out with the ACF (nothing with FFT).

So, how would you analyse this type of problem?

Bonus question: in certain cases I have missing data (1 or 2 days missing for technical reasons). How would I account for that?

PS: I am using R, but any non-R solution will do as well!

EDIT

Here is some sample data to play with: http://dl.dropbox.com/u/11676289/exampletimes.txt

Here is a plot of the times:

Plot of the times of the events

and their histogram, in 10 minutes breaks

histogram

nico
  • 4,581
  • 3
  • 32
  • 45
  • Do you have your data anywhere so people can look at it? 2. If the data is sparse enough, maybe FT (not FFT) is a good tool ($O(n_{sparse}^2)$ vs. $O(n_{all} \ln n_{all})$ ). 3. Do you care for the shape of the event or do you treat them as points / deltas with their intensity?
    • – Piotr Migdal Mar 27 '12 at 11:37
  • 1
    For the moment I don't care for the shape, I would like to make a frequency analysis first, then I will try and concentrate on shapes. I'm not sure I follow you on the FT... could you expand a little bit on that? I'll upload a sample of the data. – nico Mar 27 '12 at 12:36
  • FT - calculating it straightforwardly from its definition. This naive approach may be actually beneficial for very sparse data. – Piotr Migdal Mar 27 '12 at 13:25
  • @Piotr Migdal: ah, OK, I see – nico Mar 27 '12 at 14:01
  • Thx. I see, the date doesn't look much Fourierish. Have you tried using recurrence plots? (You said you used autocorrelation function, didn't you? I am almost sure you can see sth there, perhaps after the appropriate rescaling.) If you just care to show the burstiness see http://stats.stackexchange.com/a/20764/6552. – Piotr Migdal Mar 28 '12 at 08:46
  • @PiotrMigdal: interesting, I did not know about recurrence plot. I'll try and see if those can lead somewhere... – nico Mar 28 '12 at 08:50
  • Why mess about with FTs and such? If you want to characterize the inter-event intervals, the first step is to extract them from your data set. – Cyan Mar 30 '12 at 14:13
  • @Cyan: I've done all that standard analysis, but I got nothing really striking from it. – nico Mar 30 '12 at 16:41
  • In other words: is there a standard way to calculate the periodicity of a point process? – nico Mar 30 '12 at 16:46
  • 2
    @nico -- a quick literature search shows that work on this kind of estimation is quite recent, e.g., Shao and Lii, Modelling non-homogeneous Poisson processes with almost periodic intensity functions. I don't have access to the paper, but the authors have a presentation which summarizes their results. No available implementation that I can find... This isn't the way I'd go -- I prefer Bayesian methods. If you like, I'll <cont'd> – Cyan Apr 01 '12 at 03:01
  • 1
    ... do some EDA, and if the data looks model-able (is that a real word?) in the following fashion, I'll try to work up some R code for either log-Gaussian Cox process estimation or scaled logit-Gaussian Cox process estimation with the covariance function from Lii's previous paper on almost periodic processes that dealt with continuous state space. – Cyan Apr 01 '12 at 03:21
  • @Cyan: thank you for the references. Please write this up as an answer, even if it is not complete it gives an interesting starting point. – nico Apr 01 '12 at 05:33