COVID-19 patients elicit strong responses to the nucleocapsid (N) protein of SARS-CoV-2 but binding antibodies are also detected in prepandemic individuals, indicating potential crossreactivity with common cold human coronaviruses (HCoV) and questioning its utility in seroprevalence studies. We investigated the immunogenicity of the full-length and shorter fragments of the SARS-CoV-2 N protein, and the crossreactivity of antibodies with HCoV. We identified a C-terminus region in SARS-CoV2 N of minimal sequence homology with HCoV that was more specific for SARS-CoV-2 and highly immunogenic. IgGs to the full-length SARS-CoV-2 N also recognized N229E N, and IgGs to HKU1 N recognized SARS-CoV-2 N. Crossreactivity with SARS-CoV-2 was stronger for alpha- rather than beta-HCoV despite having less sequence identity, revealing the importance of conformational recognition. Higher preexisting IgG to OC43 N correlated with lower IgG to SARS-CoV-2 N in rRT-PCR negative individuals, reflecting less exposure and indicating a potential protective association. Antibodies to SARS-CoV-2 N were higher in patients with more severe and longer duration of symptoms and in females. IgGs remained stable for at least 3 months, while IgAs and IgMs declined faster. In conclusion, N protein is a primary target of SARS-CoV-2-specific and HCoV crossreactive antibodies, both of which may affect the acquisition of immunity to COVID-19.