The immune checkpoint inhibitors (ICIs) entered treatment algorithms in most tumors. However, the data on the efficacy is limited in rare tumors with no phase III studies. We systemically reviewed the clinical trials evaluating the ICI efficacy in rare tumors and included a total of 47 clinical trials in this review. The ICIs demonstrated over 30% response rates in Merkel cell carcinoma and squamous cell carcinoma of the skin and became the standard of care. Additionally, the ICI efficacy was promising in thymic epithelial tumors and gestational trophoblastic neoplasia. In contrast, the ICI efficacy is limited in most sarcomas, germ cell tumors and low-grade neuroendocrine tumors. The ICI efficacy seemed to be improved with combinations targeting tumor microenvironment in sarcomas. The available evidence on ICI efficacy in rare tumors denote a need for better patient selection and novel combination strategies to improve outcomes.