The transport of sea ice over the polar oceans plays an important role in climate. This transport is driven predominantly by turbulent winds, leading to stochastic motion of ice floes. Observed diffusivities and velocity distributions of sea ice deviate by orders of magnitude from Brownian models, making it challenging to predict ice transport. We fully resolve these gaps through stochastic granular simulations that account for interactions between ice floes as they respond to a noisy wind. Using only directly measured quantities as model inputs, we reproduce the dispersion, diffusivity, velocity distribution, and power spectra of sea ice observed in the Fram Strait with remarkable quantitative accuracy. We understand these features as direct consequences of collisions between floes, which rapidly dissipate the energy injected by wind. A kinetic theory provides insights into these dynamics in terms of environmental properties, and makes predictions in close agreement with observations. The ideas and tools developed here pave the way for a new predictive understanding of global sea ice transport in terms of local floe-scale processes.