Bring Metric Functions into Diffusion Models

We introduce a Cascaded Diffusion Model (Cas-DM) that improves a DenoisingDiffusion Probabilistic Model (DDPM) by effectively incorporating additionalmetric functions in training. Metric functions such as the LPIPS loss have beenproven highly effective in consistency models derived from the score matching.However, for the diffusion counterparts, the methodology and efficacy of addingextra metric functions remain unclear. One major challenge is the mismatchbetween the noise predicted by a DDPM at each step and the desired clean imagethat the metric function works well on. To address this problem, we proposeCas-DM, a network architecture that cascades two network modules to effectivelyapply metric functions to the diffusion model training. The first module,similar to a standard DDPM, learns to predict the added noise and is unaffectedby the metric function. The second cascaded module learns to predict the cleanimage, thereby facilitating the metric function computation. Experiment resultsshow that the proposed diffusion model backbone enables the effective use ofthe LPIPS loss, leading to state-of-the-art image quality (FID, sFID, IS) onvarious established benchmarks.