The Internet's rapid expansion and the current trends toward automation through intelligent systems have given malevolent software attackers a veritable playground. Numerous gadgets are effortlessly connected to the Internet, and a lot of data is being collected. Consequently, there is a growing concern about malware attacks and security threats. Malware detection has emerged as a research focus. However, there are challenges in the research, such as noise, uncertainty, and ambiguous data. The study proposes a novel framework NSDTL, that achieves state-of-the-art malware detection and classification results to address this changing threat landscape. NSDTL leverages a neutrosophic set and advanced transfer learning techniques. There are three different kinds of images in the neutrosophic domain: True (T) images, Indeterminacy (I) images, and Falsity (F) images, which deal with uncertainty. The MaleVis dataset was used for experiments on multi-class malware classification, and the findings show that NSDTL significantly outperforms current models. This study emphasizes how crucial it is to combine transfer learning with a neutrosophic set at the forefront of the continuous fight against changing cyber threats.