Forest fires can spread briskly to large-scale areas after they happen. Thus, early detection and intervention are of great importance. Unmanned aerial vehicles (UAVs) are beneficial technologies used for forest fire detection. Since flames emit very high heat and energy into their surroundings, they can be identify easily through the electro-optic infrared cameras mounted on UAVs as payloads. Detection of forest fires via UAVs has been performed by human observation in ground control stations. Convolutional Neural networks, which is effectual deep learning algorithms, are eligible for wildfire detection with UAV vision data. This paper presents a CNN based deep learning approach to the task of forest fire detection performed by human observation. We implemented state-of-art neural networks as feature extractors to the determined architecture to achieve adequate results. In the experiments, a UAV collected infrared forest fire images were used as the dataset. The experiment result clearly showed that our approach performed sufficiently on the dataset. The ResNet101-based architecture achieved the highest results in all evaluation metrics. It has confirmed itself to be the most efficient alternative with 99.20% test accuracy.