Sentiment Analysis is one of the most prominent branches of natural language processing. It deals with text classification to identify the public emotions and opinions which help businesses and political institutions make strategic decisions. This study proposes a sentiment classification model by integrating the Bayesian inference into the BERT transformer model, augmented with pre-trained GloVe embeddings. The primary objective is to refine sentiment analysis performance on IMDB movie reviews dataset by leveraging BERT's contextualized embeddings and the semantic richness of GloVe vectors, while incorporating Bayesian inference for uncertainty quantification. Using both 5-fold cross-validation and solo training, the model's performance was evaluated and produced interesting findings in both cases. The model in solo training gets an accuracy of 88.22%, an F1 score of 0.8820, and an AUC of 0.9496. Further evaluation through 5-fold cross-validation further validated the model's performance. The results indicated consistent improvement in performance across epochs, with Fold 5 reaching near-perfect classification performance (99.82% accuracy, 0.9989 AUC). This highlights the robustness of the model, as it achieved high performance across different dataset splits. The mean AUC value across all folds remained consistently high, exceeding 0.95. These results validate the efficient application of probabilistic frameworks and transformer-based models for pragmatic sentiment classification challenges in many different sectors.