The ongoing COVID-19 pandemic caused by SARS-CoV-2 is a huge public health crisis for the globe. The receptor-binding domain (RBD) of SARS-CoV-2 spike (S) protein plays a vital role in viral infection and serves as a major target for developing neutralizing antibodies. In this study, the antibody response to the RBD of SARS-CoV-2 S protein was analyzed by a panel of sera from animals immunized with RBD-based antigens and four linear B-cell epitope peptides (R345, R405, R450 and R465) were revealed. The immunogenicity of three immunodominant peptides (R345, R405, R465) was further accessed by peptide immunization in mice, and all of them could induced potent antibody response to SARS-CoV-2 S protein, indicating that the three determinants in the RBD were immunogenic. We further generated and characterized monoclonal antibodies (15G9, 12C10 and 10D2) binding to these epitope peptides, and finely mapped the three immunodominant epitopes using the corresponding antibodies. Neutralization assays showed that all three monoclonal antibodies had neutralization activity. Results from IFA and western blotting showed that 12C10 was a cross-reactive antibody against both of SARS-CoV-2 and SARS-CoV. Results from conservative and structural analysis showed that VYAWN was a highly conserved epitope and exposed on the surface of SARS-CoV-2 S trimer, whereas YQAGSTP located in the receptor binding motif (RBM) was variable among different SARS-CoV-2 strains. VRQIAP was a highly conserved, but cryptic epitope shared between SARS-CoV-2 and SARS-CoV. These findings provide important information for understanding the humoral antibody response to the RBD of SARS-CoV-2 S protein and may facilitate further efforts to design SARS-CoV-2 vaccines and the target of COVID-19 diagnostic.