With the increasing strain on the health system, there is a growing need for automatic medical image diagnosis. Emerging technologies for medical diagnosis can help to achieve the goals of sustainable development. However, analyzing medical images can be challenging due to uncertain data, ambiguity, and impreciseness. To address this issue, we have developed a novel BoneNet-NS technique to classify fractures in X-ray bone images. The proposed approach is based on the power of deep learning (DL) and neutrosophic set (NS) to deal with aleatoric uncertainty. Moreover, we present two frameworks for integrating NS with DL models: BoneNet-NS1 and BoneNet-NS2. We employ various DL models, including Xception, ResNet52V2, DenseNet121, and customized CNN to evaluate both frameworks. Furthermore, 4924 X-ray bone images are utilized to distinguish between fractured and non-fractured classes. The statistical analyses demonstrate that BoneNet-NS2 performs better than BoneNet-NS1 for most DL models. Specifically, using the ResNet52V2 model, our proposed BoneNet-NS2 achieved the highest accuracy, log loss, precision, recall, F1-score, and AUC with values of 99.7%, 0.006, 99.7%, 99.7%, 99.7%, and 99.7%, respectively.