Galaxies’ studies play an important role in the astronomic. Accurate classification of these galaxies enables scientists to understand the formation and evolution of the Universe. During the last decades, there have been several methods applied to classify the galaxy images. However, these methods encounter three big challenges. First, most existing methods converted the color images of galaxies into gray images which result in losing the essential color information. Second, the utilized feature selection methods, that used to remove the irrelevant features, may be stuck at the attractive local point. Third, using an irrelevant classifier could lead to decrease the classification accuracy. In this paper, a new algorithm is proposed to classify color images of galaxies. In this algorithm, highly accurate non-redundant color features are extracted from the color images of galaxies by using the quaternion polar complex exponential transform moments (QPCET). The quaternion representation deals with a color image in a holistic way which keeps the correlation between components and then successfully represents the color images. The QPCET moments are highly accurate, noise resistant, and numerically stable. Moreover, these moments are invariants with respect to rotation, scaling and translation (RST). These characteristics assure the excellency of the extracted color features. The Stochastic Fractal Search (SFS) has a very high ability to avoid the stuck at local point. Its binary version is utilized to select the most appropriate features which improve the classification process. The Extreme Machine learning (EML) is used to classify the color images of galaxies using the selected color features. Experiments are performed with the well-known datasets of galaxies (EFIGI catalog), where the proposed algorithm achieved high classification rate. The obtained results clearly show that the proposed method outperformed all existing galaxies classification methods.