Detector Design and Complexity Analysis for Cooperative Communications in Intersymbol Interference Fading Channels

The area of cooperative communications has recently attracted lots of research interest because of the potential benefit of increased spatial diversity. In this paper, we study the detector design for cooperative communications in intersymbol interference (ISI) channels. A novel system framework employing nonorthogonal amplify-and-forward half-duplex relays through ISI channels is introduced.We focus on detector design and first consider an optimal detector that consists of a whitening filter and a maximum-likelihood sequence estimator (MLSE). However, such an optimal detector embedded with Viterbi algorithm has practical issues of high complexity if the relay period is long. Consequently, we adopt a multitrellis Viterbi algorithm (MVA) that reduces the complexity significantly but still achieves near-optimal performance. Simulation results demonstrate the performance of both the optimal and near-optimal detector designs.