A survival analysis model for predicting time-to-total knee replacement (TKR) was developed using features from medical images and clinical measurements. Supervised and self-supervised deep learning approaches were utilized to extract features from radiographs and magnetic resonance images. Extracted features were combined with clinical and image assessments for survival analysis using random survival forests. The proposed model demonstrated high discrimination power by combining deep learning features and clinical and image assessments using a fusion of multiple modalities. The model achieved an accuracy of 75.6% and a C-Index of 84.8% for predicting the time-to-TKR surgery. Accurate time-to-TKR predictions have the potential to help assist physicians to personalize treatment strategies and improve patient outcomes.