Neighbor of BRCA1 (Nbr1) is a conserved autophagy receptor that provides cargo selectivity to autophagy. The four-tryptophan (FW) domain is a signature domain of Nbr1, but its exact function remains unclear. Here, we show that Nbr1 from the filamentous fungus Chaetomium thermophilum uses its FW domain to bind the α-mannosidase Ams1, a cargo of selective autophagy in both budding yeast and fission yeast, and delivers Ams1 to the vacuole by conventional autophagy in heterologous fission yeast. The structure of the Ams1-FW complex was determined at 2.2 Å resolution by cryo-electron microscopy. The FW domain adopts an immunoglobulin-like β-sandwich structure and recognizes the quaternary structure of the Ams1 tetramer. Notably, the N-terminal di-glycine of Ams1 is specifically recognized by a conserved pocket of the FW domain. The FW domain becomes degenerated in fission yeast Nbr1, which binds Ams1 with a ZZ domain instead. Our findings illustrate the protein binding mode of the FW domain and reveal the versatility of Nbr1-mediated cargo recognition.