In [P. Kanti, B. Kleihaus, J. Kunz, Phys. Rev. Lett. 107, 271101 (2011)] it was shown that the four-dimensional Einstein-dilaton-Gauss-Bonnet theory allows for wormholes without introducing any exotic matter. The numerical solution for the wormhole was obtained there and it was claimed that this solution is gravitationally stable against radial perturbations, what, by now, would mean the only known theoretical possibility for existence of an apparently stable, four-dimensional and asymptotically flat wormhole without exotic matter. Here, more detailed analysis of perturbations shows that the Kanti-Kleihaus-Kunz wormhole is unstable against small perturbations for any values of its parameters. The exponential growth appears in the time domain after a long period of damped oscillations, in the same way as it takes place in the case of unstable higher-dimensional black holes in the Einstein-Gauss-Bonnet theory. The instability is driven by the purely imaginary mode, which is nonperturbative in the Gauss-Bonnet coupling$\alpha$.