Fighting Fire with Fire: Adversarial Prompting to Generate a Misinformation Detection Dataset

The recent success in language generation capabilities of large languagemodels (LLMs), such as GPT, Bard, Llama etc., can potentially lead to concernsabout their possible misuse in inducing mass agitation and communal hatred viagenerating fake news and spreading misinformation. Traditional means ofdeveloping a misinformation ground-truth dataset does not scale well because ofthe extensive manual effort required to annotate the data. In this paper, wepropose an LLM-based approach of creating silver-standard ground-truth datasetsfor identifying misinformation. Specifically speaking, given a trusted newsarticle, our proposed approach involves prompting LLMs to automaticallygenerate a summarised version of the original article. The prompts in ourproposed approach act as a controlling mechanism to generate specific types offactual incorrectness in the generated summaries, e.g., incorrect quantities,false attributions etc. To investigate the usefulness of this dataset, weconduct a set of experiments where we train a range of supervised models forthe task of misinformation detection.