The recent advances in artificial intelligence modern approaches can play vital roles in the Internet of Medical Things (IoMT). Automatic diagnosis is one of the most important topics in the IoMT, including cancer diagnosis. Breast cancer is one of the top causes of death among women. Accurate diagnosis and early detection of breast cancer can improve the survival rate of patients. Deep learning models have demonstrated outstanding potential in accurately detecting and diagnosing breast cancer. This paper proposes a novel technology for breast cancer detection using CrossViT as the deep learning model and an enhanced version of the Growth Optimizer algorithm (MGO) as the feature selection method. CrossVit is a hybrid deep learning model that combines the strengths of both convolutional neural networks (CNNs) and transformers. The MGO is a meta-heuristic algorithm that selects the most relevant features from a large pool of features to enhance the performance of the model. The developed approach was evaluated on three publicly available breast cancer datasets and achieved competitive performance compared to other state-of-the-art methods. The results show that the combination of CrossViT and the MGO can effectively identify the most informative features for breast cancer detection, potentially assisting clinicians in making accurate diagnoses and improving patient outcomes. The MGO algorithm improves accuracy by approximately 1.59% on INbreast, 5.00% on MIAS, and 0.79% on MiniDDSM compared to other methods on each respective dataset. The developed approach can also be utilized to improve the Quality of Service (QoS) in the healthcare system as a deployable IoT-based intelligent solution or a decision-making assistance service, enhancing the efficiency and precision of the diagnosis.