Trace element contamination in Chinese herbal medicines has been recognized as a potential health concern for consumers. To assess the health risk to the herb-consuming population, nine trace elements (Cu, Cd, Cr, Mo, Ni, Pb, Sr, Zn, and As) were investigated based on their concentrations in three common medicinal plants (Astragalus membranaceus, Codonopsis tangshen, and Paris polyphylla var. chinensis) and soils from unpolluted and polluted areas in the Sichuan Province, China. The results showed that the metal content differed significantly in medicinal plants and soils from unpolluted versus polluted areas. No significant differences in metal accumulation were observed for these CHMs grown in either unpolluted or polluted areas. Evaluation of the health risk index suggested that soil ingestion and medicated diet represented the dominant exposure routes, indicating that trace metal(loids) in local soil might pose potential risks through soil-food chain transfer. Hazard quotient values for AM (1.473) and CT (1.357) were higher than the standard value (HQ > 1), whereas the hazard indices for PC, AM, and CT were 13.18, 14.33, and 14.01 times higher than the safe limit (HI > 1) in the polluted area, indicating non-cancer-related health hazards. Ingestion of soil was responsible for 36.39 to 91.06% of the total cancer risk and medicated diet accounted for 6.35 to 62.71%, compared with inhalation and dermal contact, suggesting carcinogenic health risks in herbs from polluted soils. In this study, Pb showed relatively higher non-carcinogenic risks, while Cr and Ni posed the highest cancer risks. Therefore, we propose more effective measures, which should be considered for Cr, Ni, and Pb remediation in soil to reduce their pollution in the studied areas.