Deep learning (DL) is transforming whole industries as complicated decision-making processes are being automated by neural networks trained on real-world data. Yet as these tools are increasingly being applied to critical problems in medicine, science, and engineering, many questions about their stability, reliability, and approximation capabilities remain. Such questions include: how many data points are sufficient to train a neural network on simple approximation tasks, and how robust are these trained architectures to noise in the data? In this work we seek to quantify the capabilities of deep neural networks (DNNs), both theoretically and numerically. Recently published results show that these architectures allow for the same convergence rates as best-in-class schemes, e.g., h,p-adaptive finite element and spectral approximations. Our own analysis confirms that DNNs afford the same sample complexity estimates as compressed sensing (CS) on sparse polynomial approximation problems. In exploring the approximation capabilities of DNNs, we also present numerical experiments on a series of simple tests in high-dimensional function approximation, with comparisons to results achieved with CS on the same problems. Our numerical experiments show that standard methods of training and initialization often yield DNNs which fail to achieve the rates of convergence suggested by theory. We conclude with a discussion of the conditioning of the DL problem.