With the rapid growth of the Internet of Things (IoT), the Internet of Medical Things (IoMT) has emerged as a critical sector that enhances convenience and plays a vital role in saving lives. IoMT devices facilitate remote access and control of various medical tools, significantly improving accessibility in the healthcare field. However, the connectivity of these devices to the internet makes them vulnerable to adversarial attacks. Safeguarding medical data becomes a paramount concern, particularly when precise biometric readings are required without compromising patient safety. This paper proposes a fast encryption mechanism to protect the color information in medical videos utilized within the IoMT environment. Our approach involves scrambling medical video frames using a rapid block-splitting method combined with simple operations. Subsequently, the scrambled frames are encrypted using different keys generated from the logistic map. To ensure the practicality of our proposed method in the IoMT setting, we implement the encryption mechanism on a cost-effective Raspberry Pi platform. To evaluate the effectiveness of our proposed mechanism, we conduct comprehensive simulations and security analyses. Notably, we investigate medical test videos during the evaluation process, further validating the applicability of our method. The results confirm our proposed mechanism's robustness by hiding patterns in original videos, achieving high entropy to increase randomness in encrypted videos, reducing the correlation between adjacent pixels in encrypted videos, and resisting various attacks.