We exploit deep combined observations with Spitzer and Chandra of the SWIRE survey in the ELAIS-N1 region, to investigate the nature of the faint X-ray and IR sources in common, to identify AGN/starburst diagnostics, and to study the sources of the X-ray and IR cosmic backgrounds. In the 17'x17' area of the Chandra ACIS-I image there are 3400 SWIRE near-IR sources with 4 sigma detections in at least 2 IRAC bands and 988 sources detected at 24micron with MIPS brighter than 0.1 mJy. Of these, 102 IRAC and 59 MIPS sources have Chandra counterparts, out of a total of 122 X-ray sources present in the area with S(0.5-8 kev)>10^(-15) erg/cm^2/s. We have constructed SEDs for each source using data from the 4 IRAC wavebands, Chandra fluxes, and optical follow-up data in the wavebands U, g', r', i', Z, and H. We fit a number of spectral templates to the SEDs at optical and infrared wavelengths to determine photometric redshifts and spectral categories, and also make use of diagnostics based on the X-ray luminosities, hardness ratios, X-ray to infrared spectral slopes and optical morphologies. Although we have spectroscopic redshifts for only a minority of the Chandra sources, the available SEDs constrain the redshifts for most of the sample sources, which turn out to be typically at 0.5<z<2. We find that 39% of the Chandra sources are dominated by type-1 AGN emission, 23% display optical/IR spectra typical of type-2 AGNs, while the remaining 38% fraction show starburst-like or even normal galaxy spectra. Since we prove that all these galaxies are dominated by AGN emission in X-rays this brings the fraction of type-1 AGNs to be 80% of the type-2: even assuming that all the Chandra sources undetected by Spitzer are type-2 AGNs, the type-1 fraction would exceed 1/3 of the total population (abridged).