Today, deep learning runs at various scales of hardware resources from the cloud and high-performance computing (HPC) centers to edge and Internet-of-Things (IoT) devices. To achieve resource-aware deep learning, we must understand the needs and challenges of deep learning applications at these different scales. In this talk, we will first investigate ways of improving hardware utilization on modern and powerful CPU-GPU co-processors, which serve as the commodity hardware for deep learning in the cloud and HPC, using workload collocation. Then, we will investigate performance and power trade-offs for deep-learning-based image analysis in space using resource-constrained edge/IoT devices.