In this study, we introduce a framework for medical image classification that combines deep learning models with modified Beluga Whale Optimization as a feature selection technique. Our approach utilizes a feature extraction technique called CrossViT, a vision transformer model architecture design of ViT. The CrossViT is used to extract relevant features from medical images. A modified version of the Beluga Whale Optimization (BWO) method is also employed to select the relevant feature. The modified BWO incorporates the Triangular Mutation Operator (TMO) approach to enhance the BWO’s ability to exploit the problem space. A set of twelve functions from the CEC2022 benchmark is used to evaluate the Modified BWO (MBWO) performance and compare it with traditional BWO, followed by assessing the proposed medical image classification framework on several benchmark datasets, demonstrating excellent performance results. This study presents an innovative and effective approach to medical image classification by combining the strengths of deep learning and metaheuristic optimization methods.