This paper presents a holistic, non-intrusive approach for drunk and drowsy detection of the driver using computer vision techniques of facial landmark detection and motion detection. The driver's continuous real-time video feed is observed with the help of a smartphone camera. A single scalar quantity, Eye Aspect Ratio (EAR) which characterizes persistent eye blinks continuously analyses this feed. Simultaneously the system checks the body and the head movements using the differential imaging technique, which operates in real-time. A severity score indicating the fitness to drive is generated cumulatively using both methods. The driver is notified with the sound of an alarm if the results are positive based on a threshold value of the severity score.