Face recognition systems are susceptible to adversarial attacks, where adversarial facial images are generated without awareness of the intrinsic attributes of the images in existing works. They change only a single attribute indiscriminately. To this end, we propose a new Semantic Adversarial Attack using StarGAN (SAA-StarGAN), which manipulates the facial attributes that are significant for each image. Specifically, we apply the cosine similarity or probability score to predict the most significant attributes. In the probability score method, we train the face verification model to perform an attribute prediction task to get a class probability score for each attribute. Then, we calculate the degree of change in the probability value in an image before and after altering the attribute. Therefore, we perform the prediction process and then alter either one or more of the most significant facial attributes under white-box or black-box settings. Experimental results illustrate that SAA-StarGAN outperforms transformation-based, gradient-based, stealthy-based, and patch-based attacks under impersonation and dodging attacks. Besides, our method achieves high attack success rates on various models in the black-box setting. In the end, the experiments confirm that the prediction of the most important attributes significantly impacts the success of adversarial attacks in both white-box and black-box settings and could improve the transferability of the generated adversarial examples.