Theory of Hallucinations based on Equivariance

This study aims to acquire knowledge for creating very large language modelsthat are immune to hallucinations. Hallucinations in contemporary largelanguage models are often attributed to a misunderstanding of real-world socialrelationships. Therefore, I hypothesize that very large language models capableof thoroughly grasping all these relationships will be free fromhallucinations. Additionally, I propose that certain types of equivariantlanguage models are adept at learning and understanding these relationships.Building on this, I have developed a specialized cross-entropy error functionto create a hallucination scale for language models, which measures theirextent of equivariance acquisition. Utilizing this scale, I tested languagemodels for their ability to acquire character-level equivariance. Inparticular, I introduce and employ a novel technique based on T5 (Text To TextTransfer Transformer) that efficiently understands permuted input texts withoutthe need for explicit dictionaries to convert token IDs (integers) to texts(strings). This T5 model demonstrated a moderate ability to acquirecharacter-level equivariance. Additionally, I discovered scale laws that canaid in developing hallucination-free language models at the character level.This methodology can be extended to assess equivariance acquisition at the wordlevel, paving the way for very large language models that can comprehensivelyunderstand relationships and, consequently, avoid hallucinations.