Genetic Instability in Cancer: An Optimal Control Problem

Carcinogenesis (cancer generation) relies on a sequence of mutations that transforms cells, leading to their unchecked growth. An important phenomenon in cancer is genetic instability, or an increased rate of mutations, with a higher associated cell death rate. Given the trade‐off between an increase in mutant cells by higher mutation rate and higher loss of mutants through a higher death rate, we ask the question: what mutation rate is most advantageous for cancer? To seek an answer to this question, we investigate an optimal control problem of normal and mutant cell growth where the abnormal mutation rate plays the role of a time‐dependent control. The analysis of this problem shows that the best “strategy” for the fastest time to cancer is to start with a high level of genetic instability initially, and then to switch to a low level of genetic instability. The exact shape of the optimal mutation rate as a function of time depends on how genetic instability contributes to the death rate of cells. If the death rate is a linear or an increasing concave function of the mutation rate, the optimal mutation rate is bang‐bang, which changes from its highest to its lowest value with a finite jump discontinuity. However, if the death rate is an increasing convex function of the mutation rate, then the optimal control is a continuous decreasing function of time. Two known mechanisms of cancer initiation have been considered, an activation of an oncogene and an inactivation of a tumor‐suppressor gene. Mathematical justification of the results of the first, a one‐step process, is reported herein.