To recognize the potential for colon polyps to develop into cancer over time, early diagnosis is crucial for preventative healthcare. Timely identification significantly improves the prognosis and treatment outcomes for colorectal cancer patients. Image segmentation is crucial in medical image analysis for accurate diagnosis and treatment planning. Therefore, in this study, we present an alternative multilevel thresholding polyp segmentation method (MPOA) to enhance the segmentation of polyp images. The proposed method is based on enhancing the planet optimization algorithm (POA) by integrating operators from the reptile search algorithm (RSA). The evaluation of the developed MPOA is tested with different polyp images and compared with other image segmentation approaches. The results highlight the superior capability of MPOA, as evidenced by various performance measures in effectively segmenting polyp images. Furthermore, metrics such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and fitness values demonstrate that MPOA outperforms the basic version of POA and other methods. The evaluation outcomes underscore the significant impact of RSA in enhancing the performance of POA for the segmentation of polyp images.