Limitations on bandwidth and power consumption impose strict bounds on data rates of diagnostic imaging systems. Consequently, the design of suitable (i.e. task- and data-aware) compression and reconstruction techniques has attracted considerable attention in recent years. Compressed sensing emerged as a popular framework for sparse signal reconstruction from a small set of compressed measurements. However, typical compressed sensing designs measure a (non)linearly weighted combination of all input signal elements, which poses practical challenges. These designs are also not necessarily task-optimal. In addition, real-time recovery is hampered by the iterative and time-consuming nature of sparse recovery algorithms. Recently, deep learning methods have shown promise for fast recovery from compressed measurements, but the design of adequate and practical sensing strategies remains a challenge. Here, we propose a deep learning solution, termed LASSY (LeArning Sub-Sampling and recoverY), that jointly learns a task-driven sub-sampling pattern and subsequent reconstruction model. The learned sub-sampling patterns are straightforwardly implementable, and based on the task at hand. LASSY's effectiveness is demonstrated in-silico for sparse signal recovery from partial Fourier measurements, and in-vivo for both anatomical-image and motion (Doppler) reconstruction from sub-sampled medical ultrasound imaging data.