Handwritten character recognition, one of the challenging problem in machine learning field and it still has got so many scopes to work on because of its huge applications. Some of that application includes, Postal address and zip code verification, writer identification, bank cheque processing and so on. Even though there are many handwritten recognition systems are developed, none gives better results when it faces more than one line of handwritten text. This work proposes a technique to improve the accuracy of handwritten recognition system. The proposed approach basically focuses on an adaptive feature extraction techniques based on HoG (Histogram of oriented gradients) and SIFT(Scale invariant feature transform).