Oxygen isotopes in marine cherts have been used to infer hot oceans during the Archean with temperatures between 60{\deg}C (333 K) and 80{\deg}C (353 K). Such climates are challenging for the early Earth warmed by the faint young Sun. The interpretation of the data has therefore been controversial. 1D climate modeling inferred that such hot climates would require very high levels of CO2 (2-6 bars). Previous carbon cycle modeling concluded that such stable hot climates were impossible and that the carbon cycle should lead to cold climates during the Hadean and the Archean. Here, we revisit the climate and carbon cycle of the early Earth at 3.8 Ga using a 3D climate-carbon model. We find that CO2 partial pressures of around 1 bar could have produced hot climates given a low land fraction and cloud feedback effects. However, such high CO2 partial pressures should not have been stable because of the weathering of terrestrial and oceanic basalts, producing an efficient stabilizing feedback. Moreover, the weathering of impact ejecta during the Late Heavy Bombardment (LHB) would have strongly reduced the CO2 partial pressure leading to cold climates and potentially snowball Earth events after large impacts. Our results therefore favor cold or temperate climates with global mean temperatures between around 8{\deg}C (281 K) and 30{\deg}C (303 K) and with 0.1-0.36 bar of CO2 for the late Hadean and early Archean. Finally, our model suggests that the carbon cycle was efficient for preserving clement conditions on the early Earth without necessarily requiring any other greenhouse gas or warming process.