The paper below describes rare de novo mutations which cause severe intellectual disability. See also Structural genomic variants (CNVs) affect cognition.
By the principle of continuity, I suspect that rare variants of smaller negative effect on cognitive ability also exist. These alleles, although harder to detect, would account for part of the observed population variation in the normal range. As discussed in an earlier post (Common variants vs mutational load), these are likely responsible for additional heritability not included in the h2 ~ 0.5 due to common variants estimated from GCTA.