Transformers originally were made for language translation. So the way I think about it, GPT models translate questions to answers. Hence the hallucinations - some questions can't be answered by just associative reasoning and pattern matching.