We propose DROGON (Deep RObust Goal-Oriented trajectory prediction Network) for accurate vehicle trajectory forecast by considering behavioral intention of vehicles in traffic scenes. Our main insight is that a causal relationship between intention and behavior of drivers can be reasoned from the observation of their relational interactions toward an environment. To succeed in causal reasoning, we build a conditional prediction model to forecast goal-oriented trajectories, which is trained with the following stages: (i) relational inference where we encode relational interactions of vehicles using the perceptual context; (ii) intention estimation to compute the probability distribution of intentional goals based on the inferred relations; and (iii) causal reasoning where we reason about the behavior of vehicles as future locations conditioned on the intention. To properly evaluate the performance of our approach, we present a new large-scale dataset collected at road intersections with diverse interactions of vehicles. The experiments demonstrate the efficacy of DROGON as it consistently outperforms state-of-the-art techniques.