Action recognition from video streams is among the active research topics in computer vision. The challenge is on the identification of the actions robustly regardless of the variations imposed by appearances of actions performed by different people. The challenge increases when the data is gathered from an outdoor environment, i.e. background and illumination variations. This paper proposes a Hidden Markov Model (HMM) based approach to model actions using Hu moments that are computed using a modified Motion History Images (MHI). The experiments performed using Weizmann dataset show that the proposed method generates 99% classification accuracy, which is comparable to many state of the art techniques that use MHI and HMMs separately.