Substitution of high-priced fish species with inexpensive alternatives and mislabeling frozen-thawed fish fillets as fresh are two important fraudulent practices of concern in the seafood industry. This study aimed to develop multimode hyperspectral imaging techniques to detect substitution and mislabeling of fish fillets. Line-scan hyperspectral images were acquired from fish fillets in four modes, including reflectance in visible and near-infrared (VNIR) region, fluorescence by 365 nm UV excitation, reflectance in short-wave infrared (SWIR) region, and Raman by 785 nm laser excitation. Fish fillets of six species (i.e., red snapper, vermilion snapper, Malabar snapper, summer flounder, white bass, and tilapia) were used for species differentiation and frozen-thawed red snapper fillets were used for freshness evaluation. All fillet samples were DNA tested to authenticate the species. A total of 24 machine learning classifiers in six categories (i.e., decision trees, discriminant analysis, Naive Bayes classifiers, support vector machines, k-nearest neighbor classifiers, and ensemble classifiers) were used for fish species and freshness classifications using four types of spectral data in three different datasets (i.e., full spectra, first ten components of principal component analysis, and bands selected by sequential feature selection method). The highest accuracies were achieved at 100% using full VNIR reflectance spectra for the species classification and 99.9% using full SWIR reflectance spectra for the freshness classification. The VNIR reflectance mode gave the overall best performance for both species and freshness inspection, and it will be further investigated as a rapid technique for detection of fish fillet substitution and mislabeling.