An automatic grading system is necessary for fresh destemmed chilli fruits. In previous works, a CNN model has been built to identify the diseased damages that appeared on the fruit’s body, and a flipping mechanism has been designed to entirely recognition by rotating the fruits. This work focussed on evaluating and implementing the CNN model to classify the diseased damaged ones on the grading model. The training data set was updated to enhance the success rate of the recognition process. Tests were carried out on 1320 fruits, in which there were 920 non-damaged fruits and 400 damaged fruits. The total testing time is 6600, and the success grading rate was found out at an average of 92%. In addition, the limitation and the causes of errors were also clarified to determine the improvement for future works.