Participants in two experiments adopted imagined perspectives in a perceptually available spatial scene and located targets by using either verbal terms (labelling) or arrows (pointing). Results revealed that performance was faster and more accurate for labelling than for pointing and more so when the adopted perspectives were misaligned with the physical orientation of the participant. This finding was obtained even when an orientation cue allowed participants to adopt the imagined perspective in advance of target presentation. A third experiment replicated these findings using traditional pointing and verbal responding. We argue that performance from misaligned perspectives in perspective-taking tasks suffers due to reference frame conflicts and that these conflicts are more pronounced with pointing and other manual responses that rely more on the reference frame of the physical body.