Breast cancer is a hazardous disease that should be seriously tackled to reduce its danger in all aspects of the world. Therefore, several imaging ways to detect this disease were considered, but the produced images need to be accurately processed to effectively detect it. Image segmentation is an indispensable step in image processing to segment the homogenous regions that have similar features such as brightness, color, texture, contrast, form, and size. Several techniques like region-based, threshold-based, edge-based, and feature-based clustering have been developed for image segmentation; however, thresholding, which is divided into two classes: bilevel and multilevel, won the highest attention by the researchers due to its simplicity, ease of use and accuracy. The multilevel thresholding-based image segmentation is difficult to be tackled using traditional techniques, especially with increasing the threshold level; therefore, the researchers pay attention to the metaheuristic algorithms which could overcome several hard problems in a reasonable time. In this paper, a new hybrid metaheuristic algorithm based on integrating the jellyfish search algorithm with an effective improvement method is proposed for segmenting the color images of breast cancer, namely the hybrid jellyfish search algorithm HJSO. Experiments are extensively performed to appear the superiority of the proposed algorithm, including validating its performance using various breast cancer images and conducting an extensive comparison with several rival algorithms to explore its effectiveness. The experimental findings, including various performance metrics like fitness values, CPU time, Peak signal-to-noise ratio (PSNR), standard deviation, Features similarity index (FSIM), and Structural similarity index (SSIM), totally show the efficiency of HJSO.