This is a theory.
We know mathematics is a recently evolved ability–all human groups can talk and compose stories, (even groups without written language,) but many groups do not even have words for numbers over three.
Out of the 200,000 years or so that anatomically modern humans have been around, no one bothered to invent written numbers until about 6,000 years ago; algebra didn’t really take off until about 1,000 years ago; calculus was invented about 400 years ago.
The ability to do any kind of abstract mathematics beyond the four basic operations is most likely a very newly acquired human skill–selection for higher math ability would have been virtually impossible prior to the invention of math, after all.
The thing about newly acquired skills is that evolution tends not to have worked out all of the kinks, yet. Things your ancestors have done consistently (including your parents) for the last 100,000 years will probably involve some decent genetic code. Things your ancestors did for millions of years will involve even better code. Things humans have been doing for only 400 years will probably involve some very kludgey code that might have some shitty side effects. Avoiding malaria comes immediately to mind–sickle cell anemia might help you avoid malaria, but it’s a pretty crappy adaptation overall. By contrast, animals have had circulatory systems for a long time; the code that builds circulatory systems is pretty solid.
Since math is a recently acquired skill, we’d expect at least some of the genes that makes people better at math to be a little, well, wonky.
It is probably no coincidence that people with extremely high math abilities have a reputation for being total weirdos, while people with very high verbal abilities–say, published authors–are regarded as pretty normal.
There are a few obvious ways to make people better at a particular task. You could take some neural real estate away from other tasks and assign it to the one you want. We know we can do this “environmentally,” (Phantom Limb Syndrome appears to be caused by the missing limb’s brain area being re-purposed for other tasks, such that when you do those other tasks, your brain simultaneously registers the activity as information coming from the limb); it seems reasonable that some sort of coding could do it genetically. Alternatively, you could increase neural speed or density or something. Or perhaps the overall size of the brain.
Each of these possibilities could also have some negative side effects–bigger brains kill mothers; re-purposing mental real estate could leave you unable to do some other function, like fine-motor control or talking.
Autism appears to basically do some set of these things (it seems to increase neural density, at least.) People with a small amount of autistic traits end up better at math than they would be otherwise. People with too many, though, suffer negative side effects–like struggling with verbal tasks. (It’s no particular surprise that autistic people tend to come from families with high math ability.) Autism is probably another mental trait that makes sense in a Sickle Cell Anemia sense.