We propose an end-to-end model to predict drug-drug interactions (DDIs) by employing graph-augmented convolutional networks. And this is implemented by combining graph CNN with an attentive pooling network to extract structural relations between drug pairs and make DDI predictions. The experiment results suggest a desirable performance achieving ROC at 0.988, F1-score at 0.956, and AUPR at 0.986. Besides, the model can tell how the two DDI drugs interact structurally by varying colored atoms. And this may be helpful for drug design during drug discovery.