Anomalies represent rare observations (e.g., data records or events) that are deviating significantly from others. Over the last forty years, researches on anomalies have received great interests because of their significance in many disciplines (e.g., computer science, chemistry, and biology). Anomaly detection, which aims to identify these rare observations, is among the most vital tasks and has shown its power in preventing detrimental events, such as financial fraud and network intrusion, from happening. The detection task is typically solved by detecting outlying data points in the features space and inherently overlooks the structural information in real-world data. Graphs have been prevalently used to preserve the structural information, and this raises the graph anomaly detection problem - identifying anomalous graph objects (i.e., nodes, edges and sub-graphs). However, conventional anomaly detection techniques cannot well solve this problem because of the complexity of graph data (e.g., irregular structures, non-independent and large-scale). For the aptitudes of deep learning in breaking these limitations, graph anomaly detection with deep learning has received intensified studies recently. In this survey, we aim to provide a systematic and comprehensive review of the contemporary deep learning techniques for graph anomaly detection. Specifically, our categorization follows a task-driven strategy and classifies existing works according to the anomalous graph objects they can detect. We especially focus on the motivations, key intuitions and technical details of existing works. We also summarize open-sourced implementations, public datasets, and commonly-used evaluation metrics for future studies. Finally, we highlight twelve future research directions according to our survey results covering emerging problems introduced by graph data, anomaly detection and real applications.