White Matter Hyperintensities (WMH) are areas of the brain that have higher intensity than other normal brain regions on Magnetic Resonance Imaging (MRI) scans. WMH is often associated with small vessel disease in the brain, making early detection of WMH important. However, there are two common issues in the detection of WMH: high ambiguity and difficulty in detecting small WMH. In this study, we propose a method called Probabilistic TransUNet to address the precision of small object segmentation and the high ambiguity of medical images. To measure model performance, we conducted a k-fold cross validation and cross dataset robustness experiment. Based on the experiments, the addition of a probabilistic model and the use of a transformer-based approach were able to achieve better performance.