The temporomandibular joint (TMJ) constitutes a bilateral ginglymoarthrodial joint, wherein each condyle interacts with its corresponding glenoid fossa of the temporal bone. There is a critical need to understand better and accurately characterize the temporomandibular joint’s diverse and variable morphological features, which can reveal significant variability across individuals, genders, and age groups. Within this study, we present an innovative condyle detection technique harnessing the potential of deep learning and feature selection (FS) models. Our approach encompasses a multi-stage process, commencing with using YOLOv8 to identify the region of interest (ROI). Subsequently, leveraging a sophisticated deep learning model, we extract salient features from the identified ROI. We modified the Energy Valley Optimizer (EVO) as an FS technique. To substantiate the efficacy of our developed method, a comprehensive dataset of 3000 panoramic images is employed, meticulously classified by two experienced maxillofacial Radiologists into four distinctive types: flat, pointed, angled, and round. The evaluation and comparison results confirm the efficiency of the proposed method in detecting condyle based on various evaluation performance indicators.