Vortices play an unique role in heat and momentum transports in astro- and geo-physics, and it is also the origin of the Earth's dynamo. A question existing for a long time is whether the movement of vortices can be predicted or understood based on their historical data. Here we use both the experiments and numerical simulations to demonstrate some generic features of vortex motion and distribution. It can be found that the vortex movement can be described on the framework of Brownian particles where they move ballistically for the time shorter than some critical timescales, and then move diffusively. Traditionally, the inertia of vortex has often been neglected when one accounts for their motion, our results imply that vortices actually have inertial-induced memory such that their short term movement can be predicted. Extending to astro- and geo-physics, the critical timescales of transition are in the order of minutes for vortices in atmosphere and ocean, in which this inertial effect may often be neglected compared to other steering sources. However, the timescales for vortices are considerably larger which range from days to a year. It infers the new concept that not only the external sources alone, for example the solar wind, but also the internal source, which is the vortex inertia, can contribute to the short term Earth's magnetic field variation.