Motivated by the success of encoding multi-scale contextual information for image analysis, we present our PointAtrousGraph (PAG) - a deep permutation-invariant hierarchical encoder-decoder architecture for learning multi-scale edge features in unorganized 3D points. Our PAG is constructed by several novel modules, such as point atrous convolution, edge-preserved pooling and edge-preserved unpooling. Similar with atrous convolution, our point atrous convolution can effectively enlarge the receptive fields of filters for learning point features without increasing computation amount. Following the idea of non-overlapping max-pooling operation, we propose our edge-preserved pooling to preserve critical edge features during subsampling. In a similar spirit, our edge-preserved unpooling propagates high-dimensional edge features while recovering the spatial information. In addition, we introduce chained skip subsampling/upsampling modules to directly propagate edge features from different hierarchies to the final stage. Experimental results show that our PAG achieves better performance compared to previous state-of-the-art methods in various applications.