For applications in population synthesis, libraries of theoretical stellar spectra are often considered an alternative to template libraries of observed spectra, because they allow a complete sampling of stellar parameters. Most attention in published theoretical spectral libraries has been devoted to the visual wavelength range. We present a detailed comparison of theoretical spectra in the range 1.57-1.67$\mu$m, for spectral types from A to early M and for giants and dwarf stars, with observed stellar spectra at resolutions around 3000, which would be sufficient to disentangle the different groups of late type stars. We have selected the NeMo grids of stellar atmospheres to perform such a comparison. We first demonstrate that after combining atomic and molecular line lists, it is possible to match observed spectral flux distributions with theoretical ones very well for almost the entire parameter range covered by the NeMo grids at moderate resolution in the visual range. In the infrared range, although the overall shape of the observed flux distributions is still matched reasonably well, the individual spectral features are reproduced by the theoretical spectra only for stars earlier than mid F type. For later spectral types the differences increase and theoretical spectra of K type stars have systematically weaker line features than those found in observations. These discrepancies are traced back to stem primarily from incomplete data on neutral atomic lines, although some of them are also related to molecules. Improving atomic data in the near infrared is a key element in making the construction of reliable libraries of stellar spectra in the infrared feasible.