Potholes can generate damage to vehicles which may lead to many accidents. Thus, accurately detecting potholes is one of the important tasks for determining proper strategies in ITS service and road management systems. Many attempts are made for developing a technology that can automatically detect and recognize potholes. One great solution is to utilize image processing techniques, as various sources of images are available, e.g., surveillance cameras, in-vehicle cameras, or smartphones. These image-based solutions help to detect and classify road damages. An image related to road damage includes objects such as potholes, shadows, lanes, and cracks, so it is hard to detect a specific object. In this paper, we propose an image-based pothole classification model. The proposed method converts RGB image data, including potholes and other objects, to grayscale to reduce computation time. It detects all objects using an object detection algorithm. We are using Gaussian and median filtering to reduce noises in the image. We use k-means clustering and the YOLO algorithm for segmentation. We are using CNN algorithm for image classification. It will show if the road has potholes or not.