We combine deep X-ray survey data from the Chandra observatory and the wide-area/shallow XMM-XXL field to estimate the AGN X-ray luminosity function in the redshift range z=3-5. The sample consists of nearly 340 sources with either photometric (212) or spectroscopic (128) redshift in the above range. The combination of deep and shallow survey fields provides a luminosity baseline of three orders of magnitude, Lx(2-10keV)~1e43-1e46erg/s at z>3. We follow a Bayesian approach to determine the binned AGN space density and explore their evolution in a model-independent way. Our methodology accounts for Poisson errors in the determination of X-ray fluxes and uncertainties in photometric redshift estimates. We demonstrate that the latter is essential for unbiased measurement of space densities. We find that the AGN X-ray luminosity function evolves strongly between the redshift intervals z=3-4 and z=4-5. There is also suggestive evidence that the amplitude of this evolution is luminosity dependent. The space density of AGN with Lx<1e45erg/s drops by a factor of 5 between the redshift intervals above, while the evolution of brighter AGN appears to be milder. Comparison of our X-ray luminosity function with that of UV/optical selected QSOs at similar redshifts shows broad agreement at bright luminosities, Lx>1e45erg/s. The faint-end slope of UV/optical luminosity functions however, is steeper than for X-ray selected AGN. This implies that the type-I AGN fraction increases with decreasing luminosity at z>3, opposite to trends established at lower redshift. We also assess the significance of AGN in keeping the hydrogen ionised at high redshift. Our X-ray luminosity function yields ionising photon rate densities that are insufficient to keep the Universe ionised at redshift z>4. A source of uncertainty in this calculation is the escape fraction of UV photons for X-ray selected AGN.