Organizational cybersecurity relies heavily on security operation centers (SOCs) to protect businesses and institutions from emerging cyber threats. In recent years, the complexity and sophistication of cyber threats have increased, pushing SOCs to their limits. As a result, SOCs struggle to address the evolving threat landscape due to their reliance on isolation technologies and reactive strategies. However, advanced technologies, such as artificial intelligence (AI) and machine learning (ML), have the potential to revolutionize SOCs by enhancing threat identification and response capabilities, as well as predicting and preempting risks. To address these challenges and highlight the full potential of SOC, this study provides a detailed overview through a comprehensive literature review that identifies gaps in existing research and examines the latest technologies used in the SOC environment to help address different operational and technical challenges and bring out their capabilities. Various methods, ranging from automated incident response and behavioral analytics to neural networks and deep learning, have been classified and compared. In addition, an in-depth reference architectural model, which is a blueprint for SOC integrating AI and ML into SOCs, is introduced. The proposed model provides a structured framework for implementation and offers insights into different SOC components and their interactions. Moreover, this systematic review emphasizes the benefits of these technologies for enhancing security operations. Finally, a case study is presented to describe the function of ML- and AI-powered SOC components to achieve optimum security. This paper concludes by discussing additional challenges and future research directions that may help advance the cybersecurity sector and provide insights into improving SOCs.