The Human activity detection is a more highly sought research as they are used for the Surveillance, Health care system, content based search, and interactive game applications. The Human activity recognition is carried out by three main stages namely object segmentation, feature extraction, their representation and human action detection using different algorithms. This paper is an attempt to identify the human activities such as running, jogging, clapping and hand waving in a video database using the method of bag of features which is the extension of bag of words. The Size Invariant Feature Transform (SIFT) based bag of SIFTs is used for the local feature extraction from the images that are chosen for the human activity detection. After finding the SIFT features for both the database and the query video, the query video features are compared with each of the database video features, the distance is found using Euclidean Distance method and the K Nearest Neighbor classifier is used to find the classified category from the list of the video . Matlab based implementation is done with various activities identified and their efficiencies are compared.