With the recent developments in technology, there has been a significant increase in the studies on analysis of human faces. Through automatic analysis of faces, it is possible to know the gender, emotional state, and even the identity of people from an image. Of them, identity or face recognition has became the most important task which has been studied for a long time now as it is crucial to take measurements for public security, credit card verification, criminal identification, and the like. In this study, we have proposed an evolutionary-based framework that relies on genetic programming algorithm to evolve a binary-and multilabel image classifier program for gender classification, facial expression recognition, and face recognition tasks. The performance of the evolved program has been compared with that of convolutional neural network, one of the most popular deep learning algorithms. The comparative results show that the proposed framework outperformed the competitor algorithm. Therefore, it has been introduced to the research community as a new binary-and multilabel image classifier framework.