Drifters designed to mimic floating marine debris and small patches of pelagic \emph{Sargassum} were satellite tracked in four regions across the North Atlantic. Though subjected to the same initial conditions at each site, the tracks of different drifters quickly diverged after deployment. We explain the clustering of drifter types using a recent Maxey-Riley theory for surface ocean inertial particle dynamics applied on multidata-based mesoscale ocean currents and winds from reanalysis. Simulated trajectories of objects at the air-sea interface are significantly improved when represented as inertial (accounting for buoyancy and size), rather than as perfectly Lagrangian (fluid following) particles. Separation distances between simulated and observed trajectories were substantially smaller for debris-like drifters than for \emph{Sargassum}-like drifters, suggesting that additional consideration of its physical properties relative to fluid velocities may be useful. Our findings can be applied to model variability in movements and distribution of diverse objects floating at the ocean surface.