In this paper, we discuss the use of live video streaming to detect RGB colors and in the process, try to track the movements of different objects in a stationary background. This color dependent tracking is achieved through MATLAB’S image processing toolbox. A live video is segmented and each video frame is used to detect red, blue or green color and keep track of the object in subsequent frames using the average shift observed in the frames. The applications of this system include robotics, color related segregation or detection, analysis and surveillance tech. devices, and security systems.