Geometric Mean Rate Maximization in RIS-aided mmWave ISAC Systems Relying on a Non-Diagonal Phase Shift Matrix

The joint optimization of the hybrid transmit precoders (HTPCs) and reflective elements of a millimeter wave (mmWave) integrated sensing and communication (ISAC) system is considered. The system also incorporates a reconfigurable intelligent surface (RIS) relying on a non-diagonal RIS (NDRIS) phase shift matrix. Specifically, we consider a hybrid architecture at the ISAC base station (BS) that serves multiple downlink communication users (CUs) via the reflected links from the RIS, while concurrently detecting multiple radar targets (RTs). We formulate an optimization problem that aims for maximizing the geometric mean (GM) rate of the CUs, subject to the sensing requirement for each RT. Additional specifications related to the limited transmit power and unit modulus (UM) constraints for both the HTPCs and the reflective elements of the NDRIS phase shift matrix make the problem challenging. To solve this problem, we first transform the intractable GM rate expression to a tractable weighted sum rate objective and next split the transformed problem into sub-problems. Consequently, we propose an iterative alternating optimization approach that leverages the majorization-minimization (MM) framework and block coordinate descent (BCD) method to solve each sub-problem. Furthermore, to tackle the UM constraints in the sub-problem of the HTPC design, we propose a penalty-based Riemannian manifold optimization (PRMO) algorithm, which optimizes the HTPCs on the Riemannian manifold. Similarly, the phases of the reflective elements of the NDRIS are optimized by employing the Riemannian manifold, and the locations of the non-zero entries of the NDRIS phase shift matrix are obtained by the maximal ratio combining (MRC) criterion. Finally, we present our simulation results, which show that deploying an NDRIS achieves additional gains for the CUs over a conventional RIS, further enhancing both the communication efficiency and sensing reliability. Furthermore, we compare the results to the pertinent benchmarks, which validate the effectiveness of our proposed algorithms.