Alzheimer’s disease (AD) is an irreversible neurodegenerative disease characterized by progressive neuronal deterioration. Early detection of AD is critical for mitigating disease progression and improving patient life. While deep learning (DL) techniques have shown promise in analyzing neuroimaging data for AD diagnosis, their interpretability has been a significant impediment toward explainable medical diagnosis. The typical practices of diagnosing AD, which involve clinical biomarkers and neuroimaging tests, have been limited in producing trustworthy and explainable progression detection models at an early stage. Another method based on finding the patient’s cognitive score through analyzing time series data has been adopted as an acceptable and cost-effective alternative in providing a deeper insight into patients’ conditions. In this study, we propose a hybrid CNN-LSTM-based model for predicting AD progression based on the fusion of four longitudinal cognitive sub-scores modalities. Our hybrid model employs the Bayesian optimizer as a computational technique to help optimize the selection of the adequate DL model’s architecture. A genetic algorithm-based feature selection has been incorporated as an optimization step to determine the best feature set from the extracted deep representations of the CNN-LSTM, and we replaced the traditional SoftMax classifier with a robust and optimized random forest classifier. An extensive set of experiments utilizing the ADNI dataset examined the operational role of each optimization step while demonstrating the effectiveness of the proposed hybrid model. The model achieved the best results compared to other DL and classical machine learning methods. To ensure diagnostic interpretability, we used the SHAP and LIME techniques to provide explainability features for the proposed model’s decisions. This work attempts to present the best possible, trustworthy decision-making AD diagnostics, potentially in deployable real-world settings.