Deep learning (DL) has proven its efficacy in extracting useful distress information from image-based data of infrastructure assets, such as pavements. Despite the overwhelming research on this topic, state-of-the-art DL approaches fail to perform satisfactorily on independent datasets as noted from an object detection-based competition. Besides, a lack of clarification in computing DL performance measures and inadequate discussion on DL implementation framework still exist. To this end, this paper contributes to the body of knowledge by synthesizing the performance of DL models from the existing relevant literature using the ‘random effect meta-analysis’ approach. Meta-analysis requires an estimate of the uncertainty in the reported performance measure (i.e., F1-score) to assign weights to individual studies and compute an overall performance measure for a group of studies. Hence, this paper introduces a statistical approach to calculate the uncertainty in the reported F1-score to compute the within-study variance. The methods, statistics, and results presented in this paper will help understand the requisites for future studies on DL in pavement distress evaluation, ultimately improving pavement asset management.