An algorithm for automated, image-based segregation of cashew kernels into different categories is the need of the hour to drive up the productivity of the Indian cashew industry. The aim of this study is to find a supervised learning model that will accurately recognize and classify the cashew kernel into different grades. Various image processing techniques are used to preprocess the cashew image dataset. K-means clustering is used to perform color image segmentation. Feature selection is performed first using neighborhood component analysis, followed by stepwise regression. Two multi-class classification methods are implemented. Support Vector Machines (SVM) with ‘one-vs-one’ classification and Adaptive Directed Acyclic Graph (ADAG) learning model showed satisfactory results. However, even higher accuracy is obtained by using the Random Forest classification model. Random Forests are easy to train, which makes them good for high dimensional data, with a large number of training examples. The main contribution of this work is developing a robust and efficient computer vision system that can grade cashew kernels on the industrial scale with high accuracy and without compromising much on the speed of computation.