The climate is a forced and dissipative nonlinear system featuring non-trivial dynamics of a vast range of spatial and temporal scales. The understanding of the climate's structural and multiscale properties is crucial for the provision of a unifying picture of its dynamics and for the implementation of accurate and efficient numerical models. We present some recent developments at the intersection between climate science, mathematics, and physics, which may prove fruitful in the direction of constructing a more comprehensive account of climate dynamics. We describe the Nambu formulation of fluid dynamics, and the potential of such a theory for constructing sophisticated numerical models of geophysical fluids. Then, we focus on the statistical mechanics of quasi-equilibrium flows in a rotating environment, which seems crucial for constructing a robust theory of geophysical turbulence. We then discuss ideas and methods suited for approaching directly the non-equilibrium nature of the climate system. First, we describe some recent findings on the thermodynamics of climate and characterize its energy and entropy budgets, and discuss related methods for intercomparing climate models and for studying tipping points. These ideas can also create a common ground between geophysics and astrophysics by suggesting general tools for studying exoplanetary atmospheres. We conclude by focusing on non-equilibrium statistical mechanics, which allows for a unified framing of problems as different as the climate response to forcings, the effect of altering the boundary conditions or the coupling between geophysical flows, and the derivation of parametrizations for numerical models.