We present photometric redshifts and spectral energy distribution (SED) classifications for a sample of 1542 optically identified sources detected with XMM in the COSMOS field. Our template fitting classifies 46 sources as stars and 464 as non-active galaxies, while the remaining 1032 require templates with an AGN contribution. High accuracy in the derived photometric redshifts was accomplished as the result of 1) photometry in up to 30 bands with high significance detections, 2) a new set of SED templates including 18 hybrids covering the far-UV to mid-infrared, which have been constructed by the combination of AGN and non-active galaxies templates, and 3) multi-epoch observations that have been used to correct for variability (most important for type 1 AGN). The reliability of the photometric redshifts is evaluated using the sub-sample of 442 sources with measured spectroscopic redshifts. We achieved an accuracy of $\sigma_{\Delta z/(1+z_{spec})} = 0.014$ for i$_{AB}^*<$22.5 ($\sigma_{\Delta z/(1+z_{spec})} \sim0.015$ for i$_{AB}^*<$24.5). The high accuracies were accomplished for both type 2 (where the SED is often dominated by the host galaxy) and type 1 AGN and QSOs out to $z=4.5$. The number of outliers is a large improvement over previous photometric redshift estimates for X-ray selected sources (4.0% and 4.8% outliers for i$_{AB}^*<$22.5 and i$_{AB}^*<$24.5, respectively). We show that the intermediate band photometry is vital to achieving accurate photometric redshifts for AGN, whereas the broad SED coverage provided by mid infrared (Spitzer/IRAC) bands is important to reduce the number of outliers for normal galaxies.