Performance evaluation of deep learning models for image analysis: Considerations for visual assessment and statistical metrics

Deep learning-based automated image analysis (DL-AIA) has been shown to outperform trained pathologists in tasks related to feature quantification. Related to these capabilities, the use of DL-AIA tools is currently extending from proof-of-principle studies to routine applications, including the evaluation of patient samples (diagnostic pathology), regulatory safety assessment (toxicologic pathology), and recurrent research tasks. To ensure that DL-AIA applications are safe and reliable, it is critical to conduct a thorough and objective generalization performance assessment to evaluate an algorithm’s ability to accurately predict patterns of interest and possibly evaluate model robustness (ie, the algorithm’s capacity to maintain predictive accuracy on images from different sources). In this article, we review the practices for performance assessment in veterinary pathology publications by which 2 main approaches were identified: (1) exclusive visual performance assessment (ie, eyeballing algorithmic predictions) plus validation of the model’s application utilizing secondary performance indices and (2) statistical performance assessment (alongside other methods), which requires creation of a test set with ground truth labels that is a hold-out from model development. This article compares the strengths and weaknesses of statistical and visual performance assessment methods. Furthermore, we discuss relevant considerations for rigorous statistical performance evaluation including metric selection, test data set image composition, ground truth label quality, resampling methods such as bootstrapping, statistical comparison of multiple models, and evaluation of model stability. It is our conclusion that visual and statistical evaluation have complementary strengths and a combined approach provides the greatest insight into the DL model’s performance and sources of error.