Prostate cancer (PCa) is one of the most common and aggressive cancers worldwide. The Gleason score (GS) system is the standard way of classifying prostate cancer and the most reliable method to determine the severity and treatment to follow. The pathologist looks at the arrangement of cancer cells in the prostate and assigns a score on a scale that ranges from 6 to 10. Automatic analysis of prostate whole-slide images (WSIs) is usually addressed as a binary classification problem, which misses the finer distinction between stages given by the GS. This paper presents a probabilistic deep learning ordinal classification method that can estimate the GS from a prostate WSI. Approaching the problem as an ordinal regression task using a differentiable probabilistic model not only improves the interpretability of the results, but also improves the accuracy of the model when compared to conventional deep classification and regression architectures.