Audio sequence transcription is a fundamental problem in Music Inforamtion Retrieval. The challenge presented by this problem is that the concurrent presence of distinct notes, played simultaneously, leads to significant distortion in the corresponding harmonics of the piano keys pressed. Proper temporal modelling is hence essential to capture the distinct pitches per frame.