Explainable artificial intelligence (XAI) is an evolving discipline that mainly emphasises unboxing in these Black-Boxes. This study provides in-depth review of XAI literature together with a new taxonomy of categorising XAI methods. Moreover, the security of Deep learning (DL) against different attacks turned to be a critical concern for both industry and academia. This study presents a taxonomic overview of the attacks on DL solutions and methods for securing DL against these attacks. Experiments are performed to evaluate and analyse the cutting-edge methods for explaining and securing DL models on real-world case studies of Twitter sentimental analysis using state-of-the-art DL models.