In March 2020, the whole world suffered from the coronavirus pandemic. This virus is a sort of virus that comes in many forms, some of which may kill. It mainly affects the human respiratory system. The development and search for COVID-19 vaccines became the global goal to stop the spread of the deadly disease. By the end of 2020, the first set of immunizations started to become available. Some countries began their immunization campaigns early. Meanwhile, others awaited the outcome of a successful trial. This research explores classifying users’ hesitation or confidence about COVID-19 immunizations. To determine the sentiment of tweets related to vaccines, we collected tweets in Arabic related to various vaccines. After collecting the tweets, we have done pre-possessing using natural language processing (NLP) techniques. After that, we developed a hybrid approach for data annotation to detect the polarity of data. We used a hybrid data annotation utilizing three different lexicons. Finally, many machine learning (ML) and deep learning (DL) methods such as Multinomial Naïve Bayes (MNB), logistic regression (LR), support vector machine (SVM), long short-term memory (LSTM), combined Gated Recurrent Unit (GRU), conventional neural network and combinations of CNN and LSTM and their hybrid versions were used and compared. Experimental results revealed that the proposed hybrid annotation method outperformed the conventional one in predicting the confidence or hesitation of people regarding COVID-19 vaccines. The maximum accuracy achieved was 98.1% using the hybrid CNN-GRU with a hybrid approach to data annotation.