Key features of immune memory are greater and faster antigen-specific antibody responses to repeat infection. In the setting of immune-evading viral evolution, it is important to understand how far antibody memory recognition stretches across viral variants when memory cells are recalled to action by repeat invasions. It is also important to understand how immune recall influences longevity of secreted antibody responses. We analyzed SARS-CoV-2 variant recognition, dynamics of memory B cells and secreted antibody over time after infection, vaccination, and boosting. We find that a two-dose SARS-CoV-2 vaccination regimen given after natural infection generated greater longitudinal antibody stability and induced maximal antibody magnitudes with enhanced breadth across Beta, Gamma, Delta and Omicron variants. A homologous 3 mRNA vaccine dose in COVID-naïve individuals conferred greater cross-variant evenness of neutralization potency with stability that was equal to the hybrid immunity conferred by infection plus vaccination. Within unvaccinated individuals who recovered from COVID, enhanced antibody stability over time was observed within a subgroup of individuals that recovered more quickly from COVID and harbored significantly more memory B cells cross-reactive to endemic coronaviruses early after infection. These cross-reactive clones map to the conserved S2 region of SARS-CoV-2 spike with higher somatic hypermutation levels and greater target affinity. We conclude that SARS-CoV-2 antigen challenge histories in humans influence not only the speed and magnitude of antibody responses, but also functional cross-variant antibody repertoire composition and longevity.