Cell types with specialized functions fundamentally regulate animal behaviour, and yet the genetic mechanisms that underlie the emergence of novel cell types and their consequences for behaviour are not well understood. Here we show that the monogamous oldfield mouse (Peromyscus polionotus) has recently evolved a novel cell type in the adrenal gland that expresses the enzyme AKR1C18, which converts progesterone into 20α-hydroxyprogesterone. We then demonstrate that 20α-hydroxyprogesterone is more abundant in oldfield mice, where it induces monogamous-typical parental behaviours, than in the closely related promiscuous deer mice (Peromyscus maniculatus). Using quantitative trait locus mapping in a cross between these species, we ultimately find interspecific genetic variation that drives expression of the nuclear protein GADD45A and the glycoprotein tenascin N, which contribute to the emergence and function of this cell type in oldfield mice. Our results provide an example by which the recent evolution of a new cell type in a gland outside the brain contributes to the evolution of social behaviour.