Automated surveillance systems are gaining importance because of their vast applications at the border while security is concerned. Various algorithms are developed and technologies are used to improve the efficiency of these surveillance systems. Efforts are being made to reduce the number of false alarms and detect any kind of suspicious activity happening in the region of suspicion within no seconds. These suspicious activities include drug smuggling, illegal immigrants crossing the borders and last but not the least, terrorist intrusion. These activities need to be detected and analyzed in order to conclude if the activity is suspicious enough to be classified as a threat. The existing systems deployed at the border are not efficient enough to detect threats and hence this paper is designed with an objective of presenting a better algorithm to make a better automated surveillance system. The sole purpose of this algorithm is to increase security at the border because safeguarding the border till date continues to remain a challenge to our country.