In this article the efficiency of marker patterns is calculated based on a new optimized automated computer-aided design system interested in nesting the irregular objects using deep learning techniques to reduce the unused area of garment layout. The automatic hyper-parameter is adjusted based on Bayesian optimization technique. Our proposed work is based on four phases: segmentation using the Grab-Cut technique which is utilized to increase our framework performance and decrease the cost. Moreover, different optimization algorithms, including k-nearest neighbor, naive Bayes, decision trees, and support vector machine are performed for classifying the pattern size from the fabric layer. Furthermore, the data augmentation technique is applied to overcome the lack of datasets and improve our framework performance by increasing the number of datasets. The optimized framework achieves 98.99% area under the receiver-operating characteristic curve, an accuracy of 98.97%, a sensitivity () of 98.98%, a precision () of 98.89%, an -score of 98.99%, a mean square error of 0.03% to 4.91%, an efficiency of 99.66%, a p-value of 0.0207, a t-statistics of 1.336, and a computational time of 3.21 s.