This study presents a new nature-inspired optimization algorithm, namely the Mantis Search Algorithm (MSA), inspired by the unique hunting behavior and sexual cannibalism of praying mantises. In brief, MSA consists of three optimization stages, including the search for prey (exploration), attack prey (exploitation), and sexual cannibalism. Those operators are simulated using various mathematical models to effectively tackle optimization challenges across diverse search spaces. The performance of MSA is rigorously tested on fifty-two optimization problems and three real-world applications (five engineering design problems, and the parameter estimation problem of photovoltaic modules and fuel cells) to show its versatility and adaptability to different scenarios. To disclose the MSA’s superiority, it is compared to two categories from the rival optimizers: the first category involves well-established and highly-cited optimizers, like Differential evolution; and the second category contains recently-published algorithms, like African Vultures Optimization Algorithm. This comparison is conducted using several performance metrics, the Wilcoxon rank-sum test and the Friedman mean rank to disclose the MSA’s effectiveness and efficiency. The results of this comparison highlight the effectiveness of this new approach and its potential for future optimization applications. The source codes of the MSA algorithm are publicly available at https://www.mathworks.com/matlabcentral/fileexchange/131833-mantis-search-algorithm-msa.