The causes of the large effect of the X chromosome in reproductive isolation and speciation have long been debated. The faster-X hypothesis predicts that X-linked loci are expected to have higher rates of adaptive evolution than autosomal loci if new beneficial mutations are on average recessive. Reproductive isolation should therefore evolve faster when contributing loci are located on the X chromosome. In this study, we have analyzed genome-wide nucleotide polymorphism data from the house mouse subspecies Mus musculus castaneus and nucleotide divergence from Mus famulus and Rattus norvegicus to compare rates of adaptive evolution for autosomal and X-linked protein-coding genes. We found significantly faster adaptive evolution for X-linked loci, particularly for genes with expression in male-specific tissues, but autosomal and X-linked genes with expression in female-specific tissues evolve at similar rates. We also estimated rates of adaptive evolution for genes expressed during spermatogenesis and found that X-linked genes that escape meiotic sex chromosome inactivation (MSCI) show rapid adaptive evolution. Our results suggest that faster-X adaptive evolution is either due to net recessivity of new advantageous mutations or due to a special gene content of the X chromosome, which regulates male function and spermatogenesis. We discuss how our results help to explain the large effect of the X chromosome in speciation.