Hybridization and invasion dynamics: insights from a spatially explicit individual-based model

Abstract

Biological invasions are a major component of global change and can substantially affect native biodiversity, community composition, and ecosystem functioning. Yet invasion success often appears inconsistent with expectations from population genetics. Introduced populations are commonly founded by relatively few individuals and are therefore expected to experience bottlenecks, reduced genetic diversity, inbreeding, and lower adaptive potential. Nevertheless, many invasive populations establish, expand, and adapt rapidly in novel environments, a contradiction known as the genetic paradox of invasions. Hybridization may help resolve this paradox by increasing genetic variation, reducing inbreeding depression, and generating novel trait combinations. However, it may also impose costs by disrupting coadapted gene complexes, generating Dobzhansky–Muller incompatibilities, or reducing local adaptation. Whether hybridization promotes or constrains invasion success is therefore likely to depend on ecological conditions and the genetic architecture of fitness-related traits, including dominance and epistasis. To investigate these processes, we developed a spatially explicit, individual-based simulation model of contact between a local and a non-native population. Implemented in R using AlphaSimR, the model simulated dispersal, mating, reproduction, and selection across a 20 × 20 landscape. It incorporated additive effects, dominance, and additive-by-additive epistasis, allowing both genetic incompatibilities and advantageous hybrid combinations to emerge. We evaluated 144 scenarios varying in epistatic effects, assortative mating, parental divergence after burn-in selection, and environmental structure. Scenarios were simulated under homogeneous conditions and across a spatial environmental gradient. For each scenario, we quantified invasion spread, hybridization, fitness, genetic and genic variance, and F ST . Across scenarios, assortative mating strongly influenced invasion and hybridization outcomes. When assortative mating was absent, extensive hybridization occurred, and invasive genotypes were often lost, whereas the local population persisted most frequently under mixed mating conditions with both assortative and random mating. Population outcomes also depended on epistatic architecture. Scenarios based on Bateson–Dobzhansky–Muller incompatibilities generally favored the persistence of local and invasive populations over hybrids, while hybrids were most successful when novel advantageous epistatic interactions emerged in hybrid genotypes. Burn-in duration further shaped outcomes: local populations persisted mainly after short burn-in phases, when parental divergence was low, whereas invasive and hybrid populations were more successful after longer burn-in phases. Overall, our results show that hybridization can either facilitate or constrain invasion success depending on mating structure, parental divergence, environmental heterogeneity, and the genetic architecture of fitness. By linking dispersal, mating system, local adaptation, and multilocus genetic mechanisms, our modeling framework provides a mechanistic basis for understanding when hybridization promotes demographic or genetic rescue and when it instead limits invasion through incompatibilities, mating barriers, or reduced hybrid fitness. More broadly, this approach highlights the value of integrating explicit population-genetic mechanisms into invasion research and helps predict the evolutionary consequences of contact between local and non-native populations.