The design of classical controllers for Voltage Source Converter High Voltage Direct Current (VSC-HVDC) transmission systems, is load-dependent and has to be adjusted for each operating condition. Thus, the robustness of such controllers becomes necessary to cope with operating condition continuous variations. Therefore, the design of hybrid optimal Artificial Intelligence Based-Sliding Mode Controllers (AI-SMCs) for VSCHVDC transmission systems is crucial research interest. These AI based controllers are proved to improve the system’s dynamic stability over a wide range of operating conditions considering different parameter variations and disturbances. For this purpose, a comprehensive state of the art of the VSC-HVDC stabilization dilemma is discussed. The nonlinear VSC-HVDC model is developed. The problem of designing a nonlinear feedback control scheme via two control strategies is addressed seeking a better performance. For ensuring robustness and chattering free behavior, the conventional SMC (C-SMC) scheme is realized using a boundary layer hyperbolic tangent function for the sliding surface. Then, the Modified Genetic Algorithm (MGA) and Particle Swarm Optimization technique (PSO) are employed for determining the optimal gains for such SMC methodology forming a modified nonlinear MGA-SMC and PSO-SMC control in order to conveniently stabilize the system and enhance its performance. The simulation results verify the enhanced performance of the VSC-HVDC transmission system controlled by both MGA-SMC and PSO-SMC compared to the C-SMC. The comparative dynamic behavior analysis for both the conventional SMC and the two meta-heuristic optimization based SMC control schemes are presented. Through simulation results, the effectiveness of the proposed metaheuristic optimization approaches and their applicability to VSC-HVDC system global stabilization and dynamic behavior enhancement are validated.