Without an effective vaccine against SARS-CoV-2, the build-up of herd immunity through natural infection has been suggested as a means to control COVID-19. Although population immunity is typically estimated by the serological investigation of recovered patients, humoral immunity in asymptomatic subjects has not been well studied, although they represent a large proportion of all SARS-CoV-2 infection cases. In this study, we conducted a serosurvey of asymptomatic infections among food workers and performed serological and cellular response analyses of asymptomatic subjects in Wuhan, the original epicenter of the COVID-19 outbreak. Our data showed that up to 5.91% of the food workers carried SARS-CoV-2 IgG antibodies asymptomatically; however, in 90.4% of them, the antibody level declined over a 2-week period. IgM and IgG antibodies, including neutralizing antibodies, were significantly lower in asymptomatic subjects than in recovered symptomatic patients with similar disease courses. Furthermore, the asymptomatic subjects showed lymphopenia and a prominent decrease in the B-cell population, as well as a low frequency of antibody-secreting cells and a low cytokine response. These factors probably contributed to the low and unsustained antibody levels in asymptomatic subjects. Our results show that asymptomatic subjects are likely to be vulnerable to SARS-CoV-2 reinfection, and neither the proportion of population immunity nor the breadth of immune responses is sufficient for herd immunity.