Accurate measurements of statistical properties, such as the star formation rate and the lifetime of young stellar objects (YSOs) in different stages, is essential for constraining star formation theories. However, it is a difficult task to separate galaxies and YSOs based on spectral energy distributions (SEDs) alone, because they contain both thermal emission from stars and dust around them and no reliable theories can be applied to distinguish them. Here we develop a machine learning algorithm based on Convolutional Neural Network, named Spectrum Classifier of Astronomical Objects (SCAO), to classify regular stars, galaxies, and YSOs, solely based on their SEDs. Superior to previous classifiers, SCAO is solely trained by labeled data without a priori theoretical knowledge, and provides excellent results with high precision (>95%) and recall (>98%) for YSOs when data from only eight bands are included. We investigate the effects of observation errors and distance effects, and show that high accuracy performance is still maintained even when using fluxes of only three bands (IRAC 3, IRAC 4, and MIPS 1) in the long wavelengths regime. We apply SCAO to Spitzer Enhanced Imaging Products (SEIP), the most complete catalog of Spitzer observations, and found 136689 YSO candidates. Finally, based on results predicted by SCAO, we provide an intuitive contour plot for a direct identification of YSOs even without any calculation. The website from SCAO is available at http://scao.astr.nthu.edu.tw.