Sex chromosome evolution is thought to be tightly associated with the acquisition and maintenance of sexual dimorphisms. Plant sex chromosomes have evolved independently in many lineages, and can provide a powerful comparative framework to study this. We assembled and annotated genome sequences of three kiwifruit species (genus Actinidia) and uncovered recurrent sex chromosome turnovers in multiple lineages. Specifically, we observed structural evolution of the neo-Y chromosomes, which was driven via rapid bursts of transposable element insertions. Surprisingly, sexual dimorphisms were conserved in the different species studied, despite the fact that the partially sex-linked genes differ between them. Using gene-editing in kiwifruit, we demonstrated that one of the two Y chromosome-encoded sex determining genes, Shy Girl, shows pleiotropic effects that can explain the conserved sexual dimorphisms. These plant sex chromosomes therefore maintain sexual dimorphisms through the conservation of a single gene, without a process involving interactions between separate sex-determining genes and genes for sexually dimorphic traits.