Applications of misspecified models in the field of survival analysis particularly frailty models may result in poor generalization and biases. Since gamma and inverse Gaussian distributions are often used interchangeably as frailty distributions for heterogeneous survival data, clear distinction between them is necessary. Based on closed form expressions of unconditional hazards and survival functions, in this paper we compare the effectiveness of gamma and inverse Gaussian distributions for frailty term in modeling survival data for heterogeneous populations. Different baseline hazards were considered including exponential, Weibull and Gompertz. We derived the closed form expressions for unconditional hazards and survival functions under each baseline distribution for both gamma and inverse Gaussian frailty models. Both graphical and extensive statistical simulation approaches are applied to compare the models. For the inference purpose, real data concerning the East Coast Fever (ECF) transmission dynamics is applied. General overview from the graphical analysis and results from both real and synthetic data indicate that gamma distribution under the Gompertz and Weibull baseline hazards is better compared to inverse Gaussian in modeling survival data for a heterogeneous population. Simulation, graphical and inferential analyses were done using appropriate packages in R language.