Ocean surface gravity waves play an important role for the air-sea momentum fluxes and the upper ocean mixing, and knowledge of the sea state leads in general circulation models to improved estimates of the ocean energy budget and allows to incorporate surface wave impacts, such as Langmuir turbulence. However, including the Stokes drift, in phase-averaged equations for the Eulerian mean motion leads to an Eulerian energy budget which is physically difficult to interpret. In this note, we show that a Lagrangian energy budget allows for a closed energy budget, in which all terms connecting the different energy compartments correspond to well known energy transfer terms. We show that the so-called Coriolis-Stokes force does not lead to an energy transfer between surface gravity waves and oceanic mean motions as previously suggested. In an energy budget for the Lagrangian mean kinetic energy, the work done by the Coriolis-Stokes force does not contribute, and should be used to estimate the kinetic energy balance in the wave affected surface mixed layer. The Lagrangian energy budget is used to discuss an energetically consistent framework, which can be used to couple a general circulation ocean model to a surface wave model.