Image processing technology has been widely used in the agricultural field. Most of it is applied to the robot that can be utilized for picking fruit and for inspection vehicles. Defect detection is a major challenge for computer vision to achieve near-human levels of recognition. The fruits and vegetable defect detection are useful in the supermarkets and can be utilized in computer vision for the automatic sorting of fruits from a set, consisting of different kinds of fruits. The objective of this work is to develop an automated tool, which can be capable of identifying and classifying mango fruits based on shape, size and color features by digital image analysis. However, defect detection by a human is labor-intensive and time-consuming. The proposed methodology is useful in supermarkets for the automatic sorting of fruits from a set of different kinds of fruits. This system minimizes error and also speeds up the time of processing. The objective of this work is to present a novel method to detect surface defects of fruit using RGB images. The proposed method uses pre-processing, segmentation, edge-detection and feature extraction to classify the fruit as defected or fresh. MATLAB has been used as the programming tool for the identification and classification of fruits using the Image Processing toolbox. The proposed method can be used to detect the visible defects, stems, size and shape of mangoes.