Digital video watermarking research has yielded numerous promising breakthroughs in recent years. As more people grow interested in videos, the copyright protection of videos has become an urgent issue that must be addressed. Digital video watermarking is a significant method for copyright protection of videos because of the growing need to safeguard video data’s intellectual property. Some existing works cannot resist geometric attacks effectively, while others fail to address the balance between robustness and visual imperceptibility. Significant progress has been made in studying fractional orthogonal moments due to their geometric invariance and beneficial image description capabilities. For these reasons, this paper proposes a robust watermarking algorithm for color video using accurate quaternion fractional Gegenbauer moments (QFrGMs), which are constructed relying on a combination of quaternion theory and accurate fractional Gegenbauer moments (FrGMs). The proposed algorithm is designed to address existing deficiencies in some related works and improve resilience. This algorithm is imperceptibly invisible and resistant to various kinds of attacks. In the proposed algorithm, the watermark information is embedded into accurately selected coefficients of QFrGMs for the selected frame of the cover video. This algorithm is robust to geometric attacks due to the outstanding geometric invariance of QFrGMs. To ensure security and improve the suggested algorithm’s security, we employed a chaotic map termed a one-dimensional Logistic Sine Cosine (LSC) map with superior chaotic characteristics to scramble the watermark. Numerical experiments were performed regarding imperceptibility and robustness to test the effectiveness. The experimental results demonstrated that the proposed method provides high visual quality and robustness against common signal processing, geometric, frame averaging, frame swapping, and frame dropping attacks. Furthermore, the proposed algorithm exhibits high robustness and good imperceptibility compared to traditional video watermarking algorithms.