The emergence and rapid spread of SARS-CoV-2, the pathogen of COVID-19, have caused a worldwide public health crisis. The SARS-CoV-2 main protease (Mpro) is an essential enzyme for the virus and therefore an appealing target for the development of antivirals to treat COVID-19 patients. Recently, many screenings have been performed against the main protease to discover novel hits. However, the actual hit rate of virtual screening is often low, and most of the predicted compounds are false positive hits. In this study, we developed a refined virtual screening strategy that incorporated molecular docking and post-docking filtering based on parameters including molecular weight and surface area, aiming to achieve predictions with fewer false positive hits. We applied this strategy to the NCI library containing 284,176 compounds against Mpro. potency analyses validated several potent inhibitors and thus confirmed the feasibility of our virtual screening strategy. Overall, The study resulted in several potent hit Mpro inhibitors, in which two inhibitors have IC values below 1 μM, that are worth being further optimized and explored. Meanwhile, the refined virtual screen strategy is also applicable to improve general screening hit rates and is useful to accelerate drug discovery for treating COVID-19 and other viral infections.