Computer vision can provide many potential advantages over manual methods of railway track inspection. Great levels of performance can be achieved through the automation of inspection using computer vision systems, as they allow scalable, quick, and cost-effective solutions to tasks otherwise unsuited to humans. At a minimum, railway track components can be objectively and quantitatively inspected, as the system does not suffer from fatigue or the subjectivity inherent with human inspectors. The digital nature of the data collection involved with a computer vision based method, archiving inspection results and trending of the data becomes feasible, leading to more advanced failure prediction models for maintenance scheduling and a more thorough understanding of railway track structure. In this research paper, a computer vision based method is presented. A system has been suggested which can periodically take images of the railway tracks and compared with the existing database of non-faulty track images on a continuous basis. If a fault arises in the track section, the system will automatically detect the fault and necessary actions can be taken, to avoid any mis-happening.