Masses are mammographic nonpalpable signs of breast cancer. These masses could be detected using screening mammography. This paper proposed a system utilizing orthogonal moment invariants (OMIs) features for mammographic masses detection and diagnosis. In this work, three sets of OMIs features were extracted. These OMIs features are Gaussian-Hermite moments (GHMs), Gegenbauer moments (GeMs), and Legendre moments (LMs). The extracted features are fused and presented to the particle swarm optimization (PSO) algorithm for feature selection. The classification step is achieved using the support vector machine (SVM). The proposed system is evaluated using 400 regions, extracted from the DDSM dataset. The obtained results reveal the promising application of OMIs features for masses detection and identification. It shows that fusing the OMIs features produces an acceptable detection performance where the area under the receiver operating characteristics (ROC) curve is Az = 0.969 ± 0.01 and the best performance of OMIs features is Az = 0.856 ± 0.053 for characterizing the malignancy of masses.