Brain age prediction based on neuroimaging data could help characterize both the typical brain development and neuropsychiatric disorders. Pattern recognition models built upon functional connectivity (FC) measures derived from resting state fMRI (rsfMRI) data have been successfully used to predict the brain age. However, most existing studies focus on coarse-grained FC measures between brain regions or intrinsic connectivity networks (ICNs), which may sacrifice fine-grained FC information of the rsfMRI data. Whole brain voxel-wise FC measures could provide fine-grained FC information of the brain and may improve the prediction performance. In this study, we develop a deep learning method to use convolutional neural networks (CNNs) to learn informative features from the fine-grained whole brain FC measures for the brain age prediction. Experimental results on a large dataset of resting-state fMRI demonstrate that the deep learning model with fine-grained FC measures could better predict the brain age.