The relative roles of evolutionary history and geographical and ecological contingency for community assembly remain unknown. Plant species, for instance, share more phytophages with closer relatives (phylogenetic conservatism), but for exotic plants introduced to another continent, this may be overlaid by geographically contingent evolution or immigration from locally abundant plant species (mass effects). We assessed within local forests to what extent exotic trees (Douglas‐fir, red oak) recruit phytophages (Coleoptera, Heteroptera) from more closely or more distantly related native plants. We found that exotics shared more phytophages with natives from the same major plant lineage (angiosperms vs. gymnosperms) than with natives from the other lineage. This was particularly true for Heteroptera, and it emphasizes the role of host specialization in phylogenetic conservatism of host use. However, for Coleoptera on Douglas‐fir, mass effects were important: immigration from beech increased with increasing beech abundance. Within a plant phylum, phylogenetic proximity of exotics and natives increased phytophage similarity, primarily in younger Coleoptera clades on angiosperms, emphasizing a role of past codiversification of hosts and phytophages. Overall, phylogenetic conservatism can shape the assembly of local phytophage communities on exotic trees. Whether it outweighs geographic contingency and mass effects depends on the interplay of phylogenetic scale, local abundance of native tree species, and the biology and evolutionary history of the phytophage taxon.