Quantitative Structure Activity Relation (QSAR) models are mathematical techniques used to link structural characteristics with biological activities, thus considered a useful tool in drug discovery, hazard evaluation, and identifying potentially lethal molecules. The QSAR regulations are determined by the Organization for Economic Cooperation and Development (OECD). QSAR models are helpful in discovering new drugs and chemicals to treat severe diseases. In order to improve the QSAR model’s predictive power for biological activities of naturally occurring indoloquinoline derivatives against different cancer cell lines, a modified machine learning (ML) technique is presented in this paper. The Arithmetic Optimization Algorithm (AOA) operators are used in the suggested model to enhance the performance of the Sinh Cosh Optimizer (SCHO). Moreover, this improvement functions as a feature selection method that eliminates superfluous descriptors. An actual dataset gathered from previously published research is utilized to evaluate the performance of the suggested model. Moreover, a comparison is made between the outcomes of the suggested model and other established methodologies. In terms of pIC50 values for different indoloquinoline derivatives against human MV4-11 (leukemia), human HCT116 (colon cancer), and human A549 (lung cancer) cell lines, the suggested model achieves root mean square error (RMSE) of 0.6822, 0.6787, 0.4411, and 0.4477, respectively. The biological application of indoloquinoline derivatives as possible anticancer medicines is predicted with a high degree of accuracy by the suggested model, as evidenced by these findings.