We introduce the framework of modular inflation with level structure, generalizing the level one theory considered previously to higher levels. We analyze the modular structure of CMB observables in this framework and show that the nontrivial geometry of the target space suffices to ensure the almost holomorphic modularity of the relevant parameters. We further introduce a concrete class of models based on hauptmodul functions that provide generators of the corresponding inflationary potentials at level $N>1$. The phenomenology of this class of models provides targets for ground-based CMB experiments in the immediate future. In the framework of our models we also discuss the status of the quantum gravity conjectures that have been formulated in the context of the swampland.