Background
The Evolution of Incest Avoidance Mechanisms

Background: Over evolutionary time, functional mechanisms are constructed through the appearance of mutations (modifications) in the designs of reproducers, which are then acted upon by selection (i.e., the impact of these modifications on the reproduction of the design). If the modification increases the number of copies of itself across generations, it will increase in frequency until it becomes universal in the species, and the species has a useful new component in its species-typical design.

Random, heritable modifications of all kinds are continually being injected into populations of organisms, so to understand the evolutionary analysis of designs, you must always ask, what would be the fate of a mutation which had property x given the other properties of the organism and the evolutionarily long-enduring features of the species' environment?
Will it spread, be eliminated, drift randomly, or what?

This question should be asked for any observed trait or hypothesized neural or psychological mechanism. It will help to evaluate the plausibility of hypotheses, and the architecture of the species under study.

Selection Pressures Against Incestuous Matings Between Close, Fertile Relatives

For example, we can ask this question about modifications that increase or decrease the sexual attraction between close relatives. What would be the fate of a mutation that made, for example, mothers sexually attractive to sons, in a world where mothers found sons attractive also? What would happen in a world where sexually mature siblings found each other erotically compelling? What would be the fate of an alternative modification that made individuals uninterested in or disgusted by sex with family members?

One can analyze such questions so as to generate expectations about the functional organization of mechanisms relating to the behavior under question.

Why Close Relatives?
Genetically, unfavorable recessive mutations are always entering into the population, every generation. These are not immediately selected out because, being recessive, they do not express themselves unless the bearer receives the same deleterious recessive from each parent. If the recessive is matched with a healthy dominant, the individual bearer is fine. If the recessive is matched with the same recessive from the other parent, the offspring expresses the unfavorable trait, and hence may die, or be functionally impaired. So, whether a child is injured by being homozygous for a given deleterious recessive is a function of the probability that both parents have the same recessive gene.

The probability that both parents will have the same rare harmful recessive is low (e.g., 1 in 1000 x 1 in 1000) unless they are related to each other. To be related means they share genes in common from a recent common ancestor. When close relatives mate, the probability that the resulting children will get paired sets of deleterious recessives jumps enormously. For example, if a lethal recessive exists in the population with a frequency of 1 in 1000, the probability an individual who gets this gene from one parent will also get it from the other is 1 in 1000. If, on the other hand, a father has the recessive, there is a 1 in 4 chance that it will pass to both his son and his daughter. If the son and daughter then mate and produce a child, the probability that they will then both pass this gene to the child is again 1 in 4 (or a total probability of being homozygous for this recessive of 1 in 16).

Thus, if an individual who is about to mate has a particular lethal recessive, the probability that it will be expressed in his child, and hence kill her or him, rises from (on the rough order of) 1 in 1000 to 1 in 16 or by a factor of 64. Humans have approximately 100,000 loci, many of which have deleterious recessives. The probability of producing a defective individual has to be summed across loci, so the real increase in impairment or lethality probabilities is much higher.

The genetic load of a species is described using the measure lethal equivalents. Most individuals have about 2 lethal equivalents, which means that if half of their loci were made homozygous, they would be dead. Thus, in humans, selection against incest between fertile, close genetic relatives is intense.

In the program, you can adjust the level of lethal equivalents. The higher the level, the more costly incest will be, and the fewer surviving descendents an individual will leave.

Adaptations for incest avoidance:
Often, in the absence of machinery for accomplishing a function, the function will not be accomplished. For example, being related to someone does not automatically generate a sexual avoidance in the absence of machinery designed to inhibit mating. For example, siblings (raised from the same parents but in different years -- non-nestlings) in certain bird species will mate as readily with each other as with any other comparable conspecific. They have evolved no method of kin-recognition for siblings raised in other years. Why should this be? When sexually mature, they disperse far enough from the nest that the probability that they will encounter a sibling is less than 1 in 1000. The cost of such a minor probability of an incestuous mating is not enough to drive the evolution of neural tissue specialized for accomplishing this function.

This illuminates two general principles:
The rarer an event in the evolutionary history of a species, the less selectively important the event will be, other things being equal. In the program you are using, you can adjust the probability an individual will encounter a relative as a potential mate. The lower the probability, the slower the selection for avoidance, and the more likely drift will take over.

Secondly, natural selection inherently involves selection between two or more alternative designs. In the absence of machinery for avoidance (or preference), one expects a lack of discrimination in mating with respect to kinship.

So, the alternatives are: being indiscriminate, being preferential (not modeled here), or rejecting mates who are closer than a given degree of kinship (e.g., siblings only; siblings and aunt/uncle/niece/nephew; cousins and closer, etc.).

So the question is, which alternatives get selected for, and which selected against.

Humans: the Westermarck mechanism

In humans, evidence indicates that incest avoidance mechanisms exist. Since genetic kinship is not a perceivable trait, the mechanism must be designed to use cues that reliably predicted kinship during human evolution. In the fusion-fission band structure of hunter-gatherers, co-residence during infancy and early childhood was such a cue. Modern humans appear to have a mechanism that decreases sexual attraction as a function of exposure during (approximately) the first 6 years of life.