Orthogonal moments (OMs) are used to extract features from color images. OMs with fractional orders are better than the OMs with integer orders due to their ability to extract fine features. This paper defined novel quaternion orthogonal shifted Gegenbauer moments (FrQSGMs) of fractional orders for color image analysis and recognition. Since both shifted Gegenbauer polynomials and the input digital images are defined in the domain [0, 1] × [0, 1], the proposed FrQSGMs did not need any image mapping or image interpolation. The invariance to geometric transformations of the proposed FrQSGMs is derived by expressing these moments in geometric moment invariants of fractional order. We conduct various experiments to test the accuracy, invariance to RST, sensitivity to noise, recognition of similar color images, and computational times. The proposed descriptors outperformed the existing orthogonal moments with fractional orders.