Severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) has rapidly spread and resulted in the global pandemic of COVID-19. Although IgM/IgG serology assay has been widely used, with the entire spike or nucleocapsid antigens, they only indicate the presence or absence of antibodies against these proteins but are not specific to the neutralization antibodies, therefore providing only generic information about infection stage and possible future immune protection. Novel technologies enabling easy-to-use and sensitive detection of multiple specific antibodies simultaneously will facilitate precise diagnosis of infection stage, prediction of clinical outcomes, and evaluation of future immune protection upon viral exposure or vaccination. Here, we demonstrate a rapid and ultrasensitive quantification method for epitope-specific antibodies, including different isotypes and subclasses, in a multiplexed manner. Using an ultrabright fluorescent nanolabel, plasmonic-fluor, this novel assay can be completed in 20 min and more importantly, the limit of detection of the plasmon-enhanced immunoassay for SARS-CoV-2 antibodies is as much as 100-fold lower compared to the assays relying on enzymatic amplification of colorimetric signals. Using convalescent patient plasma, we demonstrate that this biodetection method reveals the patient-to-patient variability in immune response as evidenced by the variations in whole protein and epitope-specific antibodies. This cost-effective, rapid, and ultrasensitive plasmonically enhanced multiplexed epitope-specific serological assay has the potential to be broadly employed in the detection of specific antibodies, which may benefit the advanced epidemiology studies and enable improvement of the clinical outcomes and prediction of the future protection against the SARS-CoV-2.