With ever-growing data and the need for developing powerful machine learning models, data owners increasingly depend on untrusted platforms (e.g., public clouds, edges, and machine learning service providers). However, sensitive data and models become susceptible to unauthorized access, misuse, and privacy compromises. Recently, a body of research has been developed to train machine learning models on encrypted outsourced data with untrusted platforms. In this survey, we summarize the studies in this emerging area with a unified framework to highlight the major challenges and approaches. We will focus on the cryptographic approaches for confidential machine learning (CML), while also covering other directions such as perturbation-based approaches and CML in the hardware-assisted confidential computing environment. The discussion will take a holistic way to consider a rich context of the related threat models, security assumptions, attacks, design philosophies, and associated trade-offs amongst data utility, cost, and confidentiality.