The graviton exchange effect on cosmological correlation functions is examined by employing the double-soft limit technique. A new relation among correlation functions that contain the effects due to graviton exchange diagrams in addition to those due to scalar-exchange and scalar-contact-interaction, is derived by using the background field method and independently by the method of Ward identities associated with dilatation symmetry. We compare these three terms, putting small values for the slow-roll parameters and $(1-n_{s}) = 0.042$, where $n_{s}$ is the scalar spectral index. It is argued that the graviton exchange effects are more dominant than the other two and could be observed in the trispectrum in the double-soft limit. Our observation strengthens the previous work by Seery, Sloth and Vernizzi, in which it has been argued that the graviton exchange dominates in the counter-collinear limit for single field slow-roll inflation.