This paper looks into the modeling of hallucination in the human's brain. Hallucinations are known to be causally associated with some malfunctions within the interaction of different areas of the brain involved in perception. Focusing on visual hallucination and its underlying causes, we identify an adversarial mechanism between different parts of the brain which are responsible in the process of visual perception. We then show how the characterized adversarial interactions in the brain can be modeled by a generative adversarial network.