Agriculture is the backbone of the country’s economy. People depend on agriculture for food and exporting to generate income. However, agriculture faces various diseases that affect the quantity and quality of vegetables. Therefore, it is important to propose a model for detecting vegetable diseases. This study proposed a sustainable smart system for vegetable disease detection and classification. This system detects early vegetable diseases in common vegetables such as tomato, potato, lettuce, and cucumber. The study employed deep learning (DL) models to detect and classify vegetable diseases. Convolutional neural networks (CNN) are a type of DL model used for image classification. This study utilizes CNN and other extensions, such as VGG16 and MobileNet, for plant image classification. Three DL models were trained on four datasets for tomato disease classification, potato disease classification, lettuce disease classification, and cucumber disease classification. The results show that the three models achieved 84.49% accuracy on the tomato disease dataset, 97.65% accuracy on the cucumber disease dataset, 97% accuracy on the potato disease dataset, and 99.9% accuracy on the lettuce disease dataset. The proposed system can assist farmers in the early detection of vegetable diseases before they spread, and it can enhance agriculture by improving both the quality and quantity of products.