In recent years, food safety incidents caused by Escherichia coli have occurred frequently, bringing a serious endangerment to human health. Due to the complex matrix of milk samples and the long pretreatment time, the existing methods cannot quickly detect E. coli in milk samples. It is necessary to enrich the E. coli in the complex matrix to improve the detection sensitivity. The E. coli outer membrane protein A (OmpA) is widely present on the cell membrane of E. coli and may be used as a new target to enrich E. coli. In this study, the purified recombinant OmpA protein was used to immunize BALB/c mice to produce polyclonal antibody. Immunomagnetic beads were combined with the polyclonal antibody to enrich the E. coli in the artificially contaminated milk samples. The products of immunoprecipitation were further used for PCR assay. The bacteria in the PCR sample can be pre-enriched, and the limit of detection is 10 × 10 cfu/mL, which is about 100 times more sensitive than samples not processed by this method. Then, the artificially contaminated milk, coffee, juice, and soybean milk samples were tested separately, and it was found that the E. coli gene could be amplified. The whole analysis time was about 120 min, including the enrichment of bacteria and the detection of eluate. We found that OmpA combined with immunomagnetic beads was more efficient, fast, and convenient than the conventional method. Bacteria can be enriched more efficiently without extracting genomic DNA and culturing bacteria. Therefore, this method has potential value for improving the detection sensitivity and shortening the detection time of E. coli in food samples.