In today's higher education institutions, predictive analytics applications have become a pressing need. In order to generate high-quality performance and valuable data for all educational levels, predictive analytics used sophisticated analytics that included the application of machine learning. the majority of people are aware that One of the main metrics that may be used by educators to track students' academic progress is their grades. In the last ten years, a wide range of machine learning algorithms has been proposed by researchers in the field of education. To improve the performance of predicting student grades, addressing imbalanced datasets presents serious difficulties. Therefore, this study gives a thorough review of machine learning algorithms to predict the final student grades in the first semester courses by enhancing the performance of prediction accuracy. In this study, we'll emphasize two modules. Using a dataset of 1282 genuine student course grades, we assess the accuracy performance of six well-known machine learning techniques: Decision Tree (J48), Support Vector Machine (SVM), Nave Bayes (NB), K-Nearest Neighbor (kNN), Logistic Regression (LR), and Random Forest (RF). In order to reduce overfitting and misclassification results brought on by imbalanced multi-classification based on oversampling Synthetic Minority Oversampling Technique (SMOTE) using two feature selection methods, we have suggested a multiclass prediction model. The outcomes demonstrate that the suggested model integrates with RF and gives a notable improvement with the greatest f-measure of 99.5%. This model's suggested findings are comparable and encouraging, and they have the potential to improve the model's performance predictions for imbalanced multi-classification for student grade prediction.