I think the correct analog of this definition in the periodic case is that coefficients $1$ through $p-1$ of the Fourier Series for $K$ all vanish.
The purpose of the definition of order is to obtain estimates of the bias of the kernel estimator. When $K$ "kills" powers $1$ through $p-1$ of $x$, then the bias will be approximately of order $h^p$ for a bandwidth $h$. This is proven in Tsybakov's Proposition 1.2 by expanding the pdf in a power series: multiplication by $K$ kills off the terms through order $p-1$, leaving the Taylor error term of order $p$; elementary estimates of that integral finish the job.
The analog of a power series for periodic functions is the Fourier Series. The analog is a perfect one: we can think of a periodic function as being defined on the unit circle in the complex plane. It has a complex coordinate $q = e^{i x}$ (where now the period is $2\pi$ rather than $1$, but that's inconsequential). Expanding $K(q)$ in a power series expresses it as a sum of powers of $q$. However, from
$$q^j = (e^{i x})^j = e^{i x j} = \cos(j x) + i \sin(j x)$$
we see that this expansion is just the Fourier Series (both the sine and cosine terms). Consequently you should be able to emulate the proof of Proposition 1.2 with very little change at all.