The musical pitch of harmonic complex sounds, such as instrumental sounds, is perceived primarily by decoding either the fundamental pitch (keynote) or spectral aspects of the stimuli, for example, single harmonics. We divided 334 professional musicians, including symphony orchestra musicians, 75 amateur musicians, and 54 nonmusicians, into either fundamental pitch listeners or spectral pitch listeners. We observed a strong correlation between pitch perception preference and asymmetry of brain structure and function in the pitch-sensitive lateral areas of Heschl's gyrus (HG), irrespective of musical ability. In particular, fundamental pitch listeners exhibited both larger gray matter volume measured using magnetic resonance imaging (MRI) and enhanced P50m activity measured using magnetoencephalography (MEG) in the left lateral HG, which is sensitive to rapid temporal processing. Their chosen instruments were percussive or high-pitched instruments that produce short, sharp, or impulsive tones (e.g., drums, guitar, piano, trumpet, or flute). By contrast, spectral pitch listeners exhibited a dominant right lateral HG, which is known to be sensitive to slower temporal and spectral processing. Their chosen instruments were lower-pitched melodic instruments that produce rather sustained tones with characteristic changes in timbre (e.g., bassoon, saxophone, french horn, violoncello, or organ). Singers also belonged to the spectral pitch listeners. Furthermore, the absolute size of the neural HG substrate depended strongly on musical ability. Overall, it is likely that both magnitude and asymmetry of lateral HG, and the related perceptual mode, may have an impact on preference for particular musical instruments and on musical performance.