Noncentral limit theorem for the generalized Hermite process

We use techniques of Malliavin calculus to study the convergence in law of a family of generalized Rosenblatt processes $Z_\gamma$ with kernels defined by parameters $\gamma$ taking values in a tetrahedral region $\Delta$ of $\RR^q$. We prove that, as $\gamma$ converges to a face of $\Delta$, the process $Z_\gamma$ converges to a compound Gaussian distribution with random variance given by the square of a Rosenblatt process of one lower rank. The convergence in law is shown to be stable. This work generalizes a previous result of Bai and Taqqu, who proved the result in the case $q=2$ and without stability.