We present a two-component Machine Learning (ML) based approach for classifying astronomical images by data-quality via an examination of sources detected in the images and image pixel values from representative sources within those images. The first component, which uses a clustering algorithm, creates a proper and small fraction of the image pixels to determine the quality of the observation. The representative images (and associated tables) are ~800 times smaller than the original images, significantly reducing the time required to train our algorithm. The useful information in the images is preserved, permitting them to be classified in different categories, but the required storage is reduced. The second component, which is a deep neural network model, classifies the representative images. Using ground-based telescope imaging data, we demonstrate that the method can be used to separate 'usable' images from those that present some problems for scientific projects -- such as images that were taken in sub-optimal conditions. This method uses two different data sets as input to a deep model and provides better performance than if we only used the images' pixel information. The method may be used in cases where large and complex data sets should be examined using deep models. Our automated classification approach achieves 97% agreement when compared to classification generated via manual image inspection. We compare our method with traditional results and show that the method improves the results by about 10%, and also presents more comprehensive outcomes.