
Model-based feature selection and clustering of RNA-seq data for unsupervised subtype discovery
Author(s) -
David K. Lim,
Naim U. Rashid,
Joseph G. Ibrahim
Publication year - 2021
Publication title -
annals of applied statistics/the annals of applied statistics
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.674
H-Index - 75
eISSN - 1941-7330
pISSN - 1932-6157
DOI - 10.1214/20-aoas1407
Subject(s) - cluster analysis , feature selection , normalization (sociology) , computer science , artificial intelligence , pattern recognition (psychology) , data mining , clustering high dimensional data , feature (linguistics) , machine learning , sociology , anthropology , linguistics , philosophy
Clustering is a form of unsupervised learning that aims to uncover latent groups within data based on similarity across a set of features. A common application of this in biomedical research is in delineating novel cancer subtypes from patient gene expression data, given a set of informative genes. However, it is typically unknown a priori what genes may be informative in discriminating between clusters, and what the optimal number of clusters are. Few methods exist for performing unsupervised clustering of RNA-seq samples, and none currently adjust for between-sample global normalization factors, select cluster-discriminatory genes, or account for potential confounding variables during clustering. To address these issues, we propose the Feature Selection and Clustering of RNA-seq (FSCseq): a model-based clustering algorithm that utilizes a finite mixture of regression (FMR) model and the quadratic penalty method with a Smoothly-Clipped Absolute Deviation (SCAD) penalty. The maximization is done by a penalized Classification EM algorithm, allowing us to include normalization factors and confounders in our modeling framework. Given the fitted model, our framework allows for subtype prediction in new patients via posterior probabilities of cluster membership, even in the presence of batch effects. Based on simulations and real data analysis, we show the advantages of our method relative to competing approaches.