In the context of the paper you are quoting, it seems the authors define a GARCH model to be linear if the past squared errors enter linearly1 into the conditional variance equation. E.g. in GARCH(1,1) we have
$$
\sigma_t^2=\omega+\alpha_1\varepsilon_{t-1}^2+\beta_1\sigma_{t-1}^2.
$$
We see that $\varepsilon_{t-1}^2$ enters linearly; $\sigma_t^2$ is obtained by multiplying $\varepsilon_{t-1}^2$ by a constant and adding some other terms. GARCH(2,2) would also be linear:
$$
\sigma_t^2=\omega+\alpha_1\varepsilon_{t-1}^2+\alpha_2\varepsilon_{t-2}^2+\beta_1\sigma_{t-1}^2+\beta_2\sigma_{t-2}^2.
$$
A nonlinear cGARCH model would have past squared error enter nonlinearly. E.g. EGARCH specifies the following:
$$
\ln(\sigma_t^2)=\omega+\alpha_1(|z_{t-1}|-\mathbb{E}(z_{t-1}))+\gamma_1 z_{t-1}+\beta_1\ln(\sigma_{t-1}^2) \quad \text{where} \quad z_{t-1}=\frac{\varepsilon_{t-1}}{\sigma_{t-1}}.
$$
Clearly, in the latter case $\sigma_t^2$ is not a linear function of past squared errors.
Other papers may define linearity in a different way, though, so it will depend on the context.
1squared errors enter linearly may sound funny, but here we take squared error as a basic unit of analysis and then ask whether it (the squared error) enters linearly or nonlinearly.