Two themes are emerging regarding the molecular genetic aetiology of intelligence. The first is that intelligence is influenced by many variants and those that are tagged by common single nucleotide polymorphisms account for around 30% of the phenotypic variation. The second, in line with other polygenic traits such as height and schizophrenia, is that these variants are not randomly distributed across the genome but cluster in genes that work together. Less clear is whether the very low range of cognitive ability (intellectual disability) is simply one end of the normal distribution describing individual differences in cognitive ability across a population. Here, we examined 40 genes with a known association with non-syndromic autosomal recessive intellectual disability (NS-ARID) to determine if they are enriched for common variants associated with the normal range of intelligence differences. The current study used the 3511 individuals of the Cognitive Ageing Genetics in England and Scotland (CAGES) consortium. In addition, a text mining analysis was used to identify gene sets biologically related to the NS-ARID set. Gene-based tests indicated that genes implicated in NS-ARID were not significantly enriched for quantitative trait loci (QTL) associated with intelligence. These findings suggest that genes in which mutations can have a large and deleterious effect on intelligence are not associated with variation across the range of intelligence differences.