Neutralizing antibody is one of the most effective interventions for acute pathogenic infection. Currently, over three million people have been identified for SARS-CoV-2 infection but SARS-CoV-2-specific vaccines and neutralizing antibodies are still lacking. SARS-CoV-2 infects host cells by interacting with angiotensin converting enzyme-2 (ACE2) via the S1 receptor-binding domain (RBD) of its surface spike glycoprotein. Therefore, blocking the interaction of SARS-CoV-2-RBD and ACE2 by antibody would cause a directly neutralizing effect against virus. In the current study, we selected the ACE2 interface of SARS-CoV-2-RBD as the targeting epitope for neutralizing antibody screening. We performed site-directed screening by phage display and finally obtained one IgG antibody (4A3) and several domain antibodies. Among them, 4A3 and three domain antibodies (4A12, 4D5, and 4A10) were identified to act as neutralizing antibodies due to their capabilities to block the interaction between SARS-CoV-2-RBD and ACE2-positive cells. The domain antibody 4A12 was predicted to have the best accessibility to all three ACE2-interfaces on the spike homotrimer. Pseudovirus and authentic SARS-CoV-2 neutralization assays showed that all four antibodies could potently protect host cells from virus infection. Overall, we isolated multiple formats of SARS-CoV-2-neutralizing antibodies via site-directed antibody screening, which could be promising candidate drugs for the prevention and treatment of COVID-19.