Sign language is one of the oldest and most natural forms of language for communication. Since most people do not know sign language and interpreters are very difficult to come by, we have come up with a real-time method using Convolution Neural Network(CNN) for fingerspelling based American sign language(ASL). In our method, the hand is first passed through a filter and after the filter has applied the hand is passed through a classifier that predicts the class of the hand gestures.