Multi‐subgroup gene screening using semi‐parametric hierarchical mixture models and the optimal discovery procedure: Application to a randomized clinical trial in multiple myeloma

Summary This article proposes an efficient approach to screening genes associated with a phenotypic variable of interest in genomic studies with subgroups. In order to capture and detect various association profiles across subgroups, we flexibly estimate the underlying effect size distribution across subgroups using a semi‐parametric hierarchical mixture model for subgroup‐specific summary statistics from independent subgroups. We then perform gene ranking and selection using an optimal discovery procedure based on the fitted model with control of false discovery rate. Efficiency of the proposed approach, compared with that based on standard regression models with covariates representing subgroups, is demonstrated through application to a randomized clinical trial with microarray gene expression data in multiple myeloma, and through a simulation experiment.