Classification of stars and galaxies is a well-known astronomical problem that has been treated using different approaches, most of them relying on morphological information. In this paper, we tackle this issue using the low-resolution spectra from narrow band photometry, provided by the PAUS (Physics of the Accelerating Universe) survey. We find that, with the photometric fluxes from the 40 narrow band filters and without including morphological information, it is possible to separate stars and galaxies to very high precision, 98.4% purity with a completeness of 98.8% for objects brighter than I = 22.5. This precision is obtained with a Convolutional Neural Network as a classification algorithm, applied to the objects' spectra. We have also applied the method to the ALHAMBRA photometric survey and we provide an updated classification for its Gold sample.