Rich phenomena from complex systems have long intrigued researchers, and yet modeling system micro-dynamics and inferring the forms of interaction are challenging for conventional data-driven approaches, being generally established by human scientists. In this study, we propose AgentNet, a generalized data-driven framework to analyze and understand the hidden interactions in complex systems. AgentNet utilizes a graph attention network to model the interaction between individual agents and employs various encoders and decoders that can be selectively applied to any desired system. Our model successfully captured three different simulated complex systems, namely cellular automata, the Vicsek model, and active Ornstein-Uhlenbeck particles in which, notably, AgentNet's visualized attention values coincided with the true interaction strength. Demonstration with empirical data from a flock of birds showed that AgentNet prediction could yield the qualitatively same collective phenomena as exhibited by real birds. We expect our framework to open a novel path to investigating complex systems and to provide insight into process-driven modeling.